Oct 30 21:48:43.072611 ip-10-0-135-7 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:48:43.505265 ip-10-0-135-7 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:48:43.505265 ip-10-0-135-7 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:48:43.505265 ip-10-0-135-7 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:48:43.505915 ip-10-0-135-7 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:48:43.508958 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.508876 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:48:43.508958 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.508958 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:48:43.509028 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.508962 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:48:43.509028 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.508965 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:48:43.509028 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.508968 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:48:43.509028 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.508971 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:48:43.509028 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.508973 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:48:43.509028 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.508976 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:48:43.509028 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.508979 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:48:43.509028 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.508983 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:48:43.509028 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.508986 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:48:43.509028 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.508988 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:48:43.509028 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.508991 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:48:43.509028 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.508994 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:48:43.509028 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.508996 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:48:43.509028 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.508999 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:48:43.509028 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509002 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:48:43.509028 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509005 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:48:43.509028 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509008 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:48:43.509028 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509010 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:48:43.509028 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509013 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:48:43.509028 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509016 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:48:43.509552 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509024 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:48:43.509552 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509028 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:48:43.509552 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509031 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:48:43.509552 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509033 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:48:43.509552 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509036 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:48:43.509552 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509041 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:48:43.509552 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509045 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:48:43.509552 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509049 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:48:43.509552 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509052 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:48:43.509552 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509055 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:48:43.509552 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509059 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:48:43.509552 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509062 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:48:43.509552 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509064 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:48:43.509552 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509067 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:48:43.509552 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509070 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:48:43.509552 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509073 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:48:43.509552 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509076 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:48:43.509552 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509079 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:48:43.509552 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509082 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:48:43.509994 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509084 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:48:43.509994 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509087 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:48:43.509994 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509090 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:48:43.509994 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509093 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:48:43.509994 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509095 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:48:43.509994 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509098 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:48:43.509994 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509101 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:48:43.509994 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509104 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:48:43.509994 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509106 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:48:43.509994 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509109 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:48:43.509994 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509111 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:48:43.509994 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509114 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:48:43.509994 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509117 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:48:43.509994 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509120 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:48:43.509994 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509123 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:48:43.509994 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509125 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:48:43.509994 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509128 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:48:43.509994 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509131 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:48:43.509994 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509133 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:48:43.509994 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509136 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:48:43.510480 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509139 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:48:43.510480 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509141 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:48:43.510480 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509144 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:48:43.510480 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509147 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:48:43.510480 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509149 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:48:43.510480 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509152 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:48:43.510480 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509155 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:48:43.510480 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509157 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:48:43.510480 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509162 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:48:43.510480 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509166 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:48:43.510480 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509169 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:48:43.510480 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509172 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:48:43.510480 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509175 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:48:43.510480 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509178 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:48:43.510480 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509180 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:48:43.510480 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509183 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:48:43.510480 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509186 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:48:43.510480 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509189 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:48:43.510480 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509193 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:48:43.510971 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509195 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:48:43.510971 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509198 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:48:43.510971 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509201 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:48:43.510971 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509204 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:48:43.510971 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509206 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:48:43.510971 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509209 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:48:43.510971 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509211 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:48:43.510971 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509215 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:48:43.510971 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509218 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:48:43.510971 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509220 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:48:43.510971 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509223 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:48:43.510971 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509226 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:48:43.510971 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509229 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:48:43.510971 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509231 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:48:43.510971 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509234 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:48:43.510971 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509237 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:48:43.510971 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509239 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:48:43.510971 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509242 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:48:43.510971 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509245 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:48:43.510971 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509260 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:48:43.511495 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509263 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:48:43.511495 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509266 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:48:43.511495 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509268 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:48:43.511495 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509271 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:48:43.511495 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509273 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:48:43.511495 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.509276 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:48:43.511495 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510075 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:48:43.511495 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510081 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:48:43.511495 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510085 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:48:43.511495 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510088 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:48:43.511495 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510091 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:48:43.511495 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510095 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:48:43.511495 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510099 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:48:43.511495 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510102 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:48:43.511495 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510105 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:48:43.511495 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510108 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:48:43.511495 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510111 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:48:43.511495 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510113 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:48:43.511495 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510116 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:48:43.511997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510119 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:48:43.511997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510122 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:48:43.511997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510125 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:48:43.511997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510128 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:48:43.511997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510131 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:48:43.511997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510134 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:48:43.511997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510137 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:48:43.511997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510139 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:48:43.511997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510142 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:48:43.511997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510145 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:48:43.511997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510147 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:48:43.511997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510150 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:48:43.511997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510152 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:48:43.511997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510155 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:48:43.511997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510158 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:48:43.511997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510160 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:48:43.511997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510163 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:48:43.511997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510166 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:48:43.511997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510168 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:48:43.512484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510171 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:48:43.512484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510173 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:48:43.512484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510177 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:48:43.512484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510179 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:48:43.512484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510182 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:48:43.512484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510185 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:48:43.512484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510188 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:48:43.512484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510191 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:48:43.512484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510194 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:48:43.512484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510197 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:48:43.512484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510199 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:48:43.512484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510202 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:48:43.512484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510205 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:48:43.512484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510208 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:48:43.512484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510211 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:48:43.512484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510214 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:48:43.512484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510216 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:48:43.512484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510219 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:48:43.512484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510221 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:48:43.512931 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510224 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:48:43.512931 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510227 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:48:43.512931 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510230 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:48:43.512931 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510234 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:48:43.512931 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510237 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:48:43.512931 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510240 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:48:43.512931 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510244 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:48:43.512931 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510260 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:48:43.512931 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510263 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:48:43.512931 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510266 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:48:43.512931 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510269 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:48:43.512931 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510272 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:48:43.512931 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510275 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:48:43.512931 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510278 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:48:43.512931 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510281 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:48:43.512931 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510284 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:48:43.512931 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510287 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:48:43.512931 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510290 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:48:43.512931 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510293 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:48:43.512931 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510295 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:48:43.513434 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510298 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:48:43.513434 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510301 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:48:43.513434 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510304 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:48:43.513434 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510307 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:48:43.513434 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510310 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:48:43.513434 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510313 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:48:43.513434 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510316 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:48:43.513434 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510319 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:48:43.513434 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510322 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:48:43.513434 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510324 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:48:43.513434 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510327 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:48:43.513434 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510330 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:48:43.513434 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510333 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:48:43.513434 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510335 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:48:43.513434 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510338 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:48:43.513434 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510341 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:48:43.513434 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510344 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:48:43.513434 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510346 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:48:43.513434 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510349 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:48:43.513434 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510352 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:48:43.513920 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510354 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:48:43.513920 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510357 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:48:43.513920 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510360 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:48:43.513920 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510362 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:48:43.513920 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510365 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:48:43.513920 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510367 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:48:43.513920 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510370 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:48:43.513920 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510373 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:48:43.513920 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510376 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:48:43.513920 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510378 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:48:43.513920 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510381 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:48:43.513920 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510385 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:48:43.513920 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510388 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:48:43.513920 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510390 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:48:43.513920 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.510394 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:48:43.513920 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511127 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:48:43.513920 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511136 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:48:43.513920 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511140 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:48:43.513920 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511144 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:48:43.514396 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511149 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:48:43.514396 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511153 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:48:43.514396 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511157 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:48:43.514396 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511160 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:48:43.514396 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511164 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:48:43.514396 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511173 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:48:43.514396 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511177 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:48:43.514396 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511181 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:48:43.514396 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511212 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:48:43.514396 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511216 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:48:43.514396 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511220 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:48:43.514396 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511224 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:48:43.514396 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511229 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:48:43.514396 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511233 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:48:43.514396 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511237 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:48:43.514396 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511241 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:48:43.514396 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511262 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:48:43.514396 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511268 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:48:43.514396 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511272 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:48:43.514847 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511277 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:48:43.514847 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511281 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:48:43.514847 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511286 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:48:43.514847 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511290 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:48:43.514847 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511295 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:48:43.514847 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511301 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:48:43.514847 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511306 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:48:43.514847 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511311 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:48:43.514847 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511318 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:48:43.514847 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511322 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:48:43.514847 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511326 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:48:43.514847 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511332 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:48:43.514847 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511335 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:48:43.514847 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511339 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:48:43.514847 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511344 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:48:43.514847 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511348 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:48:43.514847 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511352 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:48:43.514847 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511355 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:48:43.514847 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511360 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:48:43.514847 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511364 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:48:43.515349 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511368 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:48:43.515349 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511372 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:48:43.515349 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511376 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:48:43.515349 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511380 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:48:43.515349 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511384 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:48:43.515349 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511388 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:48:43.515349 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511392 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:48:43.515349 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511396 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:48:43.515349 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511400 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:48:43.515349 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511404 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:48:43.515349 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511408 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:48:43.515349 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511412 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:48:43.515349 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511417 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:48:43.515349 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511422 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:48:43.515349 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511426 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:48:43.515349 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511431 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:48:43.515349 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511436 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:48:43.515349 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511440 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:48:43.515349 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511445 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:48:43.515349 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511449 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:48:43.515833 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511453 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:48:43.515833 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511459 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:48:43.515833 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511463 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:48:43.515833 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511468 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:48:43.515833 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511472 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:48:43.515833 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511476 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:48:43.515833 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511480 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:48:43.515833 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511485 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:48:43.515833 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511489 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:48:43.515833 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511494 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:48:43.515833 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511497 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:48:43.515833 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511502 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:48:43.515833 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511507 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:48:43.515833 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511511 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:48:43.515833 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511515 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:48:43.515833 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511519 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:48:43.515833 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511523 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:48:43.515833 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511527 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:48:43.515833 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511531 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:48:43.516281 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511535 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:48:43.516281 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511540 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:48:43.516281 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511546 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:48:43.516281 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511552 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:48:43.516281 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511557 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:48:43.516281 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511562 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:48:43.516281 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511566 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:48:43.516281 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511570 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:48:43.516281 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511575 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:48:43.516281 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511580 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:48:43.516281 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511584 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:48:43.516281 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511588 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:48:43.516281 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511591 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:48:43.516281 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511596 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:48:43.516281 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511600 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:48:43.516281 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511605 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:48:43.516281 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511609 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:48:43.516281 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511613 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:48:43.516281 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511617 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:48:43.516738 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511621 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:48:43.516738 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511625 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:48:43.516738 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511630 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:48:43.516738 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511634 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:48:43.516738 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.511637 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:48:43.516738 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512807 2578 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:48:43.516738 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512823 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:48:43.516738 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512830 2578 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:48:43.516738 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512835 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:48:43.516738 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512840 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:48:43.516738 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512843 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:48:43.516738 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512848 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:48:43.516738 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512852 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:48:43.516738 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512855 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:48:43.516738 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512859 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:48:43.516738 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512862 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:48:43.516738 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512866 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:48:43.516738 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512869 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:48:43.516738 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512872 2578 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:48:43.516738 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512875 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:48:43.516738 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512878 2578 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:48:43.516738 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512881 2578 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:48:43.516738 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512884 2578 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:48:43.517307 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512890 2578 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:48:43.517307 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512893 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:48:43.517307 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512896 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:48:43.517307 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512900 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:48:43.517307 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512904 2578 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:48:43.517307 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512908 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:48:43.517307 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512912 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:48:43.517307 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512915 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:48:43.517307 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512919 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:48:43.517307 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512922 2578 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:48:43.517307 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512925 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:48:43.517307 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512928 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:48:43.517307 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512932 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:48:43.517307 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512935 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:48:43.517307 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512940 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:48:43.517307 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512943 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:48:43.517307 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512946 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:48:43.517307 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512949 2578 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:48:43.517307 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512953 2578 flags.go:64] FLAG: --enable-server="true" Oct 30 21:48:43.517307 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512956 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:48:43.517307 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512961 2578 flags.go:64] FLAG: --event-burst="100" Oct 30 21:48:43.517307 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512964 2578 flags.go:64] FLAG: --event-qps="50" Oct 30 21:48:43.517307 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512967 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:48:43.517307 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512970 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:48:43.517933 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512974 2578 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:48:43.517933 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512978 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:48:43.517933 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512982 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:48:43.517933 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512985 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:48:43.517933 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512988 2578 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:48:43.517933 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512992 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:48:43.517933 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512995 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:48:43.517933 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.512998 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:48:43.517933 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513001 2578 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:48:43.517933 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513004 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:48:43.517933 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513007 2578 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:48:43.517933 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513010 2578 flags.go:64] FLAG: --feature-gates="" Oct 30 21:48:43.517933 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513014 2578 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:48:43.517933 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513017 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:48:43.517933 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513020 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:48:43.517933 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513024 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:48:43.517933 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513027 2578 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:48:43.517933 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513030 2578 flags.go:64] FLAG: --help="false" Oct 30 21:48:43.517933 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513033 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:48:43.517933 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513037 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:48:43.517933 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513040 2578 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:48:43.517933 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513043 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:48:43.517933 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513046 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:48:43.518488 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513050 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:48:43.518488 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513053 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:48:43.518488 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513056 2578 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:48:43.518488 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513059 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:48:43.518488 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513062 2578 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:48:43.518488 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513065 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:48:43.518488 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513068 2578 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:48:43.518488 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513071 2578 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:48:43.518488 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513075 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:48:43.518488 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513078 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:48:43.518488 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513081 2578 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:48:43.518488 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513084 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:48:43.518488 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513087 2578 flags.go:64] FLAG: --lock-file="" Oct 30 21:48:43.518488 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513090 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:48:43.518488 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513093 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:48:43.518488 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513096 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:48:43.518488 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513102 2578 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:48:43.518488 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513105 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:48:43.518488 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513108 2578 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:48:43.518488 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513111 2578 flags.go:64] FLAG: --logging-format="text" Oct 30 21:48:43.518488 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513114 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:48:43.518488 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513117 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:48:43.518488 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513120 2578 flags.go:64] FLAG: --manifest-url="" Oct 30 21:48:43.518488 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513124 2578 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:48:43.518488 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513129 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:48:43.519095 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513132 2578 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:48:43.519095 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513137 2578 flags.go:64] FLAG: --max-pods="110" Oct 30 21:48:43.519095 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513140 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:48:43.519095 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513143 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:48:43.519095 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513149 2578 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:48:43.519095 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513152 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:48:43.519095 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513156 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:48:43.519095 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513159 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:48:43.519095 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513163 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:48:43.519095 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513171 2578 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:48:43.519095 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513174 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:48:43.519095 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513177 2578 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:48:43.519095 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513180 2578 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:48:43.519095 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513183 2578 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:48:43.519095 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513186 2578 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:48:43.519095 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513189 2578 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:48:43.519095 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513192 2578 flags.go:64] FLAG: --port="10250" Oct 30 21:48:43.519095 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513196 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:48:43.519095 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513199 2578 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-06c6e410af1c4c253" Oct 30 21:48:43.519095 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513202 2578 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:48:43.519095 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513205 2578 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:48:43.519095 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513208 2578 flags.go:64] FLAG: --register-node="true" Oct 30 21:48:43.519095 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513211 2578 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:48:43.519095 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513214 2578 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:48:43.519095 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513217 2578 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:48:43.519095 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513220 2578 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:48:43.519903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513223 2578 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:48:43.519903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513227 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:48:43.519903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513230 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:48:43.519903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513234 2578 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:48:43.519903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513236 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:48:43.519903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513240 2578 flags.go:64] FLAG: --runonce="false" Oct 30 21:48:43.519903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513243 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:48:43.519903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513259 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:48:43.519903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513265 2578 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:48:43.519903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513269 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:48:43.519903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513274 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:48:43.519903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513278 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:48:43.519903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513281 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:48:43.519903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513286 2578 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:48:43.519903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513289 2578 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:48:43.519903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513292 2578 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:48:43.519903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513295 2578 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:48:43.519903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513298 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:48:43.519903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513301 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:48:43.519903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513304 2578 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:48:43.519903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513307 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:48:43.519903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513313 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:48:43.519903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513316 2578 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:48:43.519903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513319 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:48:43.519903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513324 2578 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:48:43.520991 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513327 2578 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:48:43.520991 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513330 2578 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:48:43.520991 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513333 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:48:43.520991 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513336 2578 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:48:43.520991 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513339 2578 flags.go:64] FLAG: --v="2" Oct 30 21:48:43.520991 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513344 2578 flags.go:64] FLAG: --version="false" Oct 30 21:48:43.520991 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513348 2578 flags.go:64] FLAG: --vmodule="" Oct 30 21:48:43.520991 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513352 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:48:43.520991 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513356 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:48:43.520991 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513498 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:48:43.520991 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513503 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:48:43.520991 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513507 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:48:43.520991 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513510 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:48:43.520991 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513513 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:48:43.520991 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513515 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:48:43.520991 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513518 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:48:43.520991 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513521 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:48:43.520991 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513526 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:48:43.520991 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513529 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:48:43.520991 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513531 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:48:43.520991 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513535 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:48:43.520991 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513538 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:48:43.521724 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513541 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:48:43.521724 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513544 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:48:43.521724 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513546 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:48:43.521724 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513549 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:48:43.521724 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513552 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:48:43.521724 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513554 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:48:43.521724 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513558 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:48:43.521724 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513560 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:48:43.521724 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513563 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:48:43.521724 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513566 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:48:43.521724 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513568 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:48:43.521724 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513571 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:48:43.521724 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513574 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:48:43.521724 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513577 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:48:43.521724 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513580 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:48:43.521724 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513582 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:48:43.521724 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513585 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:48:43.521724 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513588 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:48:43.521724 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513591 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:48:43.521724 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513593 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:48:43.522534 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513596 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:48:43.522534 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513598 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:48:43.522534 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513601 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:48:43.522534 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513604 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:48:43.522534 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513606 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:48:43.522534 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513609 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:48:43.522534 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513613 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:48:43.522534 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513617 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:48:43.522534 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513620 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:48:43.522534 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513622 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:48:43.522534 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513626 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:48:43.522534 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513629 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:48:43.522534 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513631 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:48:43.522534 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513634 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:48:43.522534 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513637 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:48:43.522534 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513639 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:48:43.522534 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513642 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:48:43.522534 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513645 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:48:43.522534 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513647 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:48:43.523133 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513650 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:48:43.523133 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513652 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:48:43.523133 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513655 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:48:43.523133 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513658 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:48:43.523133 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513660 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:48:43.523133 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513663 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:48:43.523133 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513665 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:48:43.523133 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513669 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:48:43.523133 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513674 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:48:43.523133 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513677 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:48:43.523133 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513680 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:48:43.523133 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513683 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:48:43.523133 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513686 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:48:43.523133 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513689 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:48:43.523133 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513691 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:48:43.523133 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513694 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:48:43.523133 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513697 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:48:43.523133 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513699 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:48:43.523133 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513702 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:48:43.523613 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513705 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:48:43.523613 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513709 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:48:43.523613 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513712 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:48:43.523613 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513715 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:48:43.523613 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513719 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:48:43.523613 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513721 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:48:43.523613 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513724 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:48:43.523613 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513726 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:48:43.523613 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513729 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:48:43.523613 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513732 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:48:43.523613 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513734 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:48:43.523613 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513737 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:48:43.523613 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513740 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:48:43.523613 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513742 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:48:43.523613 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513745 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:48:43.523613 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513747 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:48:43.523613 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513750 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:48:43.523613 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513753 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:48:43.523613 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513755 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:48:43.523613 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513758 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:48:43.524096 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513761 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:48:43.524096 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513763 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:48:43.524096 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513766 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:48:43.524096 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513769 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:48:43.524096 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513771 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:48:43.524096 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513774 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:48:43.524096 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513776 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:48:43.524096 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513779 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:48:43.524096 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513782 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:48:43.524096 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513784 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:48:43.524096 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513788 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:48:43.524096 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513792 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:48:43.524096 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513795 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:48:43.524096 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513799 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:48:43.524096 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513803 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:48:43.524096 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.513830 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:48:43.524504 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513868 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:48:43.524504 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513874 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:48:43.524504 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513877 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:48:43.524504 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513880 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:48:43.524504 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513883 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:48:43.524504 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513886 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:48:43.524504 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513889 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:48:43.524504 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513891 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:48:43.524504 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513894 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:48:43.524504 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513897 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:48:43.524504 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513899 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:48:43.524504 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513902 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:48:43.524504 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513904 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:48:43.524504 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513907 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:48:43.524504 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513910 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:48:43.524504 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513912 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:48:43.524504 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513915 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:48:43.524504 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513918 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:48:43.524504 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513920 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:48:43.525034 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513923 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:48:43.525034 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513926 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:48:43.525034 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513929 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:48:43.525034 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513931 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:48:43.525034 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513935 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:48:43.525034 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513939 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:48:43.525034 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513942 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:48:43.525034 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513945 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:48:43.525034 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513948 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:48:43.525034 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513950 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:48:43.525034 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513953 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:48:43.525034 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513957 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:48:43.525034 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513960 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:48:43.525034 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513964 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:48:43.525034 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513967 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:48:43.525034 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513969 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:48:43.525034 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513972 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:48:43.525034 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513975 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:48:43.525034 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513978 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:48:43.525518 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513980 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:48:43.525518 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513983 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:48:43.525518 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513985 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:48:43.525518 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513988 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:48:43.525518 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513991 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:48:43.525518 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513993 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:48:43.525518 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513996 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:48:43.525518 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.513999 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:48:43.525518 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514002 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:48:43.525518 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514005 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:48:43.525518 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514007 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:48:43.525518 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514010 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:48:43.525518 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514012 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:48:43.525518 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514015 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:48:43.525518 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514018 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:48:43.525518 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514020 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:48:43.525518 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514023 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:48:43.525518 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514025 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:48:43.525518 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514028 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:48:43.525518 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514031 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:48:43.526009 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514033 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:48:43.526009 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514036 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:48:43.526009 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514038 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:48:43.526009 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514041 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:48:43.526009 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514045 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:48:43.526009 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514047 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:48:43.526009 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514052 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:48:43.526009 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514056 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:48:43.526009 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514060 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:48:43.526009 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514063 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:48:43.526009 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514067 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:48:43.526009 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514069 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:48:43.526009 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514072 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:48:43.526009 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514075 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:48:43.526009 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514078 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:48:43.526009 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514081 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:48:43.526009 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514084 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:48:43.526009 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514086 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:48:43.526009 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514089 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:48:43.526009 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514091 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:48:43.526505 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514094 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:48:43.526505 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514096 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:48:43.526505 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514099 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:48:43.526505 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514102 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:48:43.526505 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514105 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:48:43.526505 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514107 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:48:43.526505 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514110 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:48:43.526505 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514112 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:48:43.526505 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514115 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:48:43.526505 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514117 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:48:43.526505 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514120 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:48:43.526505 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514123 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:48:43.526505 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514125 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:48:43.526505 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514128 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:48:43.526505 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514130 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:48:43.526505 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514133 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:48:43.526505 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514137 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:48:43.526505 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514139 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:48:43.526505 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514144 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:48:43.526505 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514148 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:48:43.527003 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514150 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:48:43.527003 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514153 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:48:43.527003 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514156 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:48:43.527003 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514158 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:48:43.527003 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514161 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:48:43.527003 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514163 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:48:43.527003 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514166 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:48:43.527003 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.514168 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:48:43.527003 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.523985 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:48:43.527003 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.524010 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:48:43.527003 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.524027 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:48:43.527003 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.524033 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:48:43.527003 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524100 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:48:43.527003 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524106 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:48:43.527003 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524109 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:48:43.527003 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524113 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:48:43.527003 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524117 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:48:43.527003 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524119 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:48:43.527003 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524122 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:48:43.527475 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524125 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:48:43.527475 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524127 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:48:43.527475 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524130 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:48:43.527475 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524133 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:48:43.527475 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524136 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:48:43.527475 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524139 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:48:43.527475 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524142 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:48:43.527475 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524145 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:48:43.527475 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524147 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:48:43.527475 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524151 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:48:43.527475 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524154 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:48:43.527475 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524157 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:48:43.527475 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524160 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:48:43.527475 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524163 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:48:43.527475 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524166 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:48:43.527475 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524169 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:48:43.527475 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524171 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:48:43.527475 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524174 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:48:43.527475 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524177 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:48:43.527964 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524180 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:48:43.527964 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524183 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:48:43.527964 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524186 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:48:43.527964 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524189 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:48:43.527964 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524192 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:48:43.527964 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524195 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:48:43.527964 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524197 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:48:43.527964 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524202 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:48:43.527964 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524206 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:48:43.527964 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524210 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:48:43.527964 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524213 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:48:43.527964 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524217 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:48:43.527964 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524220 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:48:43.527964 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524223 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:48:43.527964 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524226 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:48:43.527964 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524228 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:48:43.527964 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524231 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:48:43.527964 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524234 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:48:43.527964 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524236 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:48:43.528484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524239 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:48:43.528484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524241 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:48:43.528484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524244 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:48:43.528484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524263 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:48:43.528484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524267 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:48:43.528484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524270 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:48:43.528484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524273 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:48:43.528484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524276 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:48:43.528484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524278 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:48:43.528484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524281 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:48:43.528484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524285 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:48:43.528484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524288 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:48:43.528484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524291 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:48:43.528484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524294 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:48:43.528484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524296 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:48:43.528484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524299 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:48:43.528484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524302 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:48:43.528484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524304 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:48:43.528484 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524308 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:48:43.528943 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524312 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:48:43.528943 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524315 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:48:43.528943 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524318 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:48:43.528943 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524321 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:48:43.528943 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524324 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:48:43.528943 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524326 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:48:43.528943 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524329 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:48:43.528943 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524332 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:48:43.528943 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524334 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:48:43.528943 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524337 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:48:43.528943 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524340 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:48:43.528943 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524343 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:48:43.528943 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524345 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:48:43.528943 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524348 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:48:43.528943 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524351 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:48:43.528943 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524360 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:48:43.528943 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524363 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:48:43.528943 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524366 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:48:43.528943 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524368 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:48:43.528943 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524371 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:48:43.529440 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524374 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:48:43.529440 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524376 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:48:43.529440 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524379 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:48:43.529440 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524381 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:48:43.529440 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524384 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:48:43.529440 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524387 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:48:43.529440 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524389 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:48:43.529440 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524392 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:48:43.529440 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524395 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:48:43.529440 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524397 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:48:43.529440 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524400 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:48:43.529440 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524403 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:48:43.529440 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524406 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:48:43.529440 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524409 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:48:43.529440 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524411 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:48:43.529440 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524414 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:48:43.529440 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524417 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:48:43.529440 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524419 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:48:43.529440 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524422 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:48:43.529440 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524424 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:48:43.529906 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524427 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:48:43.529906 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524430 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:48:43.529906 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.524458 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:48:43.529906 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524618 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:48:43.529906 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524623 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:48:43.529906 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524626 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:48:43.529906 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524629 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:48:43.529906 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524632 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:48:43.529906 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524640 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:48:43.529906 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524643 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:48:43.529906 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524647 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:48:43.529906 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524650 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:48:43.529906 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524653 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:48:43.529906 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524656 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:48:43.529906 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524659 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:48:43.530270 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524661 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:48:43.530270 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524664 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:48:43.530270 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524667 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:48:43.530270 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524671 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:48:43.530270 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524676 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:48:43.530270 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524679 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:48:43.530270 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524681 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:48:43.530270 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524685 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:48:43.530270 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524688 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:48:43.530270 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524691 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:48:43.530270 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524693 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:48:43.530270 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524696 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:48:43.530270 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524699 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:48:43.530270 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524702 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:48:43.530270 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524704 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:48:43.530270 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524707 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:48:43.530270 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524710 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:48:43.530270 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524712 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:48:43.530270 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524715 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:48:43.530725 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524718 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:48:43.530725 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524720 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:48:43.530725 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524723 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:48:43.530725 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524727 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:48:43.530725 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524730 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:48:43.530725 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524734 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:48:43.530725 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524737 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:48:43.530725 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524747 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:48:43.530725 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524750 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:48:43.530725 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524753 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:48:43.530725 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524755 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:48:43.530725 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524758 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:48:43.530725 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524760 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:48:43.530725 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524763 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:48:43.530725 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524766 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:48:43.530725 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524768 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:48:43.530725 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524771 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:48:43.530725 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524774 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:48:43.530725 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524776 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:48:43.531218 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524779 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:48:43.531218 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524781 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:48:43.531218 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524784 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:48:43.531218 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524787 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:48:43.531218 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524789 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:48:43.531218 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524792 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:48:43.531218 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524794 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:48:43.531218 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524796 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:48:43.531218 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524799 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:48:43.531218 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524802 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:48:43.531218 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524804 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:48:43.531218 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524806 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:48:43.531218 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524809 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:48:43.531218 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524812 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:48:43.531218 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524814 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:48:43.531218 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524817 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:48:43.531218 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524820 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:48:43.531218 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524823 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:48:43.531218 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524825 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:48:43.531218 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524828 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:48:43.531701 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524831 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:48:43.531701 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524839 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:48:43.531701 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524842 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:48:43.531701 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524845 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:48:43.531701 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524847 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:48:43.531701 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524850 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:48:43.531701 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524853 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:48:43.531701 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524855 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:48:43.531701 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524858 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:48:43.531701 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524860 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:48:43.531701 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524863 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:48:43.531701 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524865 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:48:43.531701 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524868 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:48:43.531701 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524870 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:48:43.531701 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524873 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:48:43.531701 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524876 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:48:43.531701 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524879 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:48:43.531701 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524881 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:48:43.531701 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524884 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:48:43.531701 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524886 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:48:43.532199 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524889 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:48:43.532199 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524892 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:48:43.532199 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524895 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:48:43.532199 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524897 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:48:43.532199 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524900 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:48:43.532199 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524903 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:48:43.532199 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524926 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:48:43.532199 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524930 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:48:43.532199 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524933 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:48:43.532199 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524936 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:48:43.532199 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524940 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:48:43.532199 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524943 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:48:43.532199 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524946 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:48:43.532199 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524949 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:48:43.532199 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524957 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:48:43.532199 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.524960 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:48:43.532597 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.524983 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:48:43.532597 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525024 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:48:43.532597 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525028 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:48:43.532597 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525031 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:48:43.532597 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525034 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:48:43.532597 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525039 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:48:43.532597 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525042 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:48:43.532597 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525045 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:48:43.532597 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525047 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:48:43.532597 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525050 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:48:43.532597 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525053 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:48:43.532597 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525055 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:48:43.532597 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525058 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:48:43.532597 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525061 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:48:43.532597 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525063 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:48:43.532597 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525066 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:48:43.532975 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525069 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:48:43.532975 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525071 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:48:43.532975 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525074 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:48:43.532975 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525077 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:48:43.532975 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525079 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:48:43.532975 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525082 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:48:43.532975 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525084 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:48:43.532975 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525087 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:48:43.532975 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525089 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:48:43.532975 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525092 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:48:43.532975 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525095 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:48:43.532975 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525097 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:48:43.532975 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525100 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:48:43.532975 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525103 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:48:43.532975 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525106 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:48:43.532975 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525114 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:48:43.532975 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525119 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:48:43.532975 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525123 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:48:43.533426 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525126 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:48:43.533426 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525128 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:48:43.533426 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525131 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:48:43.533426 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525135 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:48:43.533426 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525138 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:48:43.533426 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525140 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:48:43.533426 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525143 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:48:43.533426 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525146 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:48:43.533426 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525148 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:48:43.533426 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525151 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:48:43.533426 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525154 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:48:43.533426 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525157 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:48:43.533426 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525160 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:48:43.533426 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525162 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:48:43.533426 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525165 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:48:43.533426 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525167 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:48:43.533426 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525170 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:48:43.533426 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525173 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:48:43.533426 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525178 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:48:43.533997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525181 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:48:43.533997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525183 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:48:43.533997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525186 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:48:43.533997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525188 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:48:43.533997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525191 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:48:43.533997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525193 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:48:43.533997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525196 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:48:43.533997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525199 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:48:43.533997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525201 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:48:43.533997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525204 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:48:43.533997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525206 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:48:43.533997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525214 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:48:43.533997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525217 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:48:43.533997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525220 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:48:43.533997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525222 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:48:43.533997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525225 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:48:43.533997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525229 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:48:43.533997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525231 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:48:43.533997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525234 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:48:43.533997 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525236 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:48:43.534579 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525239 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:48:43.534579 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525241 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:48:43.534579 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525244 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:48:43.534579 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525260 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:48:43.534579 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525265 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:48:43.534579 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525268 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:48:43.534579 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525270 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:48:43.534579 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525273 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:48:43.534579 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525276 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:48:43.534579 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525280 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:48:43.534579 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525284 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:48:43.534579 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525288 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:48:43.534579 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525292 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:48:43.534579 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525295 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:48:43.534579 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525299 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:48:43.534579 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525303 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:48:43.534579 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525307 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:48:43.534579 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525311 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:48:43.534579 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525315 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:48:43.534579 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525319 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:48:43.535035 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525323 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:48:43.535035 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525329 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:48:43.535035 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525333 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:48:43.535035 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525337 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:48:43.535035 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525349 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:48:43.535035 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525353 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:48:43.535035 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525357 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:48:43.535035 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525361 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:48:43.535035 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525366 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:48:43.535035 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525370 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:48:43.535035 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525374 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:48:43.535035 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525377 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:48:43.535035 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525381 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:48:43.535035 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:43.525385 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:48:43.535035 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.525504 2578 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:48:43.535035 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.528659 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 21:48:43.535035 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.529740 2578 server.go:1014] "Starting client certificate rotation" Oct 30 21:48:43.535035 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.529844 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:48:43.535035 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.530453 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:48:43.535774 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.531993 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:48:43.535774 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.535150 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:48:43.543395 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.543376 2578 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:48:43.547370 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.547353 2578 log.go:25] "Validated CRI v1 image API" Oct 30 21:48:43.548274 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.548229 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:48:43.551048 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.551028 2578 fs.go:135] Filesystem UUIDs: map[3578952b-c51b-4da5-bbc6-ad5ddc55ef30:/dev/nvme0n1p4 49414ccb-8903-4060-bec4-bc8797a77cc7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 21:48:43.551141 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.551047 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 21:48:43.557067 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.556943 2578 manager.go:217] Machine: {Timestamp:2026-10-30 21:48:43.555410744 +0000 UTC m=+0.375144291 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3086220 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2456eceec74055364afe40911ad25f SystemUUID:ec2456ec-eec7-4055-364a-fe40911ad25f BootID:12542b92-67c7-4e79-bcd5-56086dc469da Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:e4:5d:83:d7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:e4:5d:83:d7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:4d:ed:24:92:8e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:48:43.557750 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.557736 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:48:43.557869 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.557854 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:48:43.561351 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.561327 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:48:43.561486 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.561353 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-7.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:48:43.561532 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.561491 2578 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:48:43.561532 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.561499 2578 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:48:43.561532 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.561509 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:48:43.561532 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.561520 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:48:43.562754 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.562742 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:48:43.569503 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.569482 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:48:43.569663 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.569653 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:48:43.570481 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.570471 2578 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:48:43.570529 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.570485 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:48:43.570529 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.570496 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:48:43.570529 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.570506 2578 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:48:43.570529 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.570514 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:48:43.570881 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.570868 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:48:43.574165 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.574142 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:48:43.581923 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.581903 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:48:43.582961 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.582946 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:48:43.583016 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.582984 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:48:43.583241 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.583227 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:48:43.583293 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.583263 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:48:43.583293 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.583274 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:48:43.583293 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.583282 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:48:43.583293 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.583290 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:48:43.583394 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.583298 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:48:43.583813 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.583803 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:48:43.583841 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.583816 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:48:43.583841 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.583828 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:48:43.583841 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.583837 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:48:43.583916 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.583856 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:48:43.583916 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.583869 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:48:43.584817 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.584801 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:48:43.584817 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.584816 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:48:43.590130 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.590111 2578 server.go:1263] "Started kubelet" Oct 30 21:48:43.590294 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.590260 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:48:43.590390 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.590292 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:48:43.590390 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.590351 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:48:43.590887 ip-10-0-135-7 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:48:43.591903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.591874 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:48:43.593732 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.593715 2578 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:48:43.597075 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:43.597056 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 21:48:43.599136 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.599120 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:48:43.599200 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.599135 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:48:43.599658 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:43.599506 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-7.us-west-2.compute.internal\" not found" Oct 30 21:48:43.599878 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.599858 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:48:43.599878 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.599860 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:48:43.600045 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.600032 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:48:43.600138 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.600124 2578 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:48:43.600138 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.600138 2578 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:48:43.600593 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.600393 2578 factory.go:153] Registering CRI-O factory Oct 30 21:48:43.600593 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.600410 2578 factory.go:223] Registration of the crio container factory successfully Oct 30 21:48:43.600593 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.600461 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:48:43.600593 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.600472 2578 factory.go:55] Registering systemd factory Oct 30 21:48:43.600593 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.600481 2578 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:48:43.600593 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.600497 2578 factory.go:103] Registering Raw factory Oct 30 21:48:43.600593 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.600512 2578 manager.go:1199] Started watching for new ooms in manager Oct 30 21:48:43.601858 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.601841 2578 manager.go:319] Starting recovery of all containers Oct 30 21:48:43.614889 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.614870 2578 manager.go:324] Recovery completed Oct 30 21:48:43.618757 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.618744 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:48:43.620593 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.620576 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:48:43.620658 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.620606 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:48:43.620658 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.620616 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:48:43.621139 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.621125 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:48:43.621222 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.621125 2578 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:48:43.621222 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.621180 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:48:43.621222 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.621209 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:48:43.621805 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.621790 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:48:43.621880 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.621821 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:48:43.621880 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.621836 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:48:43.623361 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.623349 2578 policy_none.go:50] "Start" Oct 30 21:48:43.623425 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.623364 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:48:43.623425 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.623374 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:48:43.625364 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.625348 2578 policy_none.go:44] "Start" Oct 30 21:48:43.678834 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.670885 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:48:43.678834 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.675774 2578 manager.go:341] "Starting Device Plugin manager" Oct 30 21:48:43.678834 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:43.675989 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:48:43.678834 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.676002 2578 server.go:87] "Starting device plugin registration server" Oct 30 21:48:43.678834 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.676216 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:48:43.678834 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.676280 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:48:43.678834 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.676292 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:48:43.678834 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.676503 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:48:43.678834 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.676579 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:48:43.678834 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.676588 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:48:43.678834 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:43.677195 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:48:43.678834 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:43.677265 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-7.us-west-2.compute.internal\" not found" Oct 30 21:48:43.678834 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.677802 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:48:43.678834 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.678443 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:48:43.678834 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.678494 2578 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:48:43.678834 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:43.678537 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 21:48:43.736378 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.736332 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-7.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 21:48:43.736378 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:43.736336 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-7.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 21:48:43.737522 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:43.736320 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-7.us-west-2.compute.internal.18e36d000bdd6ab9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-7.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-7.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:48:43.590077113 +0000 UTC m=+0.409810661,LastTimestamp:2026-10-30 21:48:43.590077113 +0000 UTC m=+0.409810661,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-7.us-west-2.compute.internal,}" Oct 30 21:48:43.776449 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.776374 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:48:43.777360 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.777340 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:48:43.777475 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.777371 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:48:43.777475 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.777384 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:48:43.777475 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.777409 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:48:43.780062 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.780044 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal"] Oct 30 21:48:43.780115 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.780104 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:48:43.780874 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.780854 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:48:43.780993 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.780888 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:48:43.780993 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.780903 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:48:43.783086 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.783073 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:48:43.783211 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.783196 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:48:43.783283 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.783229 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:48:43.783773 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.783755 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:48:43.783773 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.783765 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:48:43.783884 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.783786 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:48:43.783884 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.783797 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:48:43.783884 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.783787 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:48:43.783884 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.783875 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:48:43.785941 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.785928 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:48:43.786003 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.785960 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:48:43.786683 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.786668 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:48:43.786770 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.786690 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:48:43.786770 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.786700 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:48:43.794130 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:43.794051 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-7.us-west-2.compute.internal.18e36d000daf0a8d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-7.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-7.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-7.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:48:43.620592269 +0000 UTC m=+0.440325816,LastTimestamp:2026-10-30 21:48:43.620592269 +0000 UTC m=+0.440325816,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-7.us-west-2.compute.internal,}" Oct 30 21:48:43.794275 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:43.794158 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 21:48:43.794275 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:43.794167 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-7.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 21:48:43.794275 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:43.794196 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 21:48:43.801131 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.801112 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/556c8f6eb09b355e5322f215128e2cca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal\" (UID: \"556c8f6eb09b355e5322f215128e2cca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:48:43.801204 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.801139 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/556c8f6eb09b355e5322f215128e2cca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal\" (UID: \"556c8f6eb09b355e5322f215128e2cca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:48:43.801204 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.801155 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/774172d5b7cfcf0d4b41c09fc8c830a1-config\") pod \"kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal\" (UID: \"774172d5b7cfcf0d4b41c09fc8c830a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:48:43.816157 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:43.816129 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-7.us-west-2.compute.internal\" not found" node="ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:48:43.818811 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:43.818790 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-7.us-west-2.compute.internal\" not found" node="ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:48:43.822105 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.822085 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:48:43.822183 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.822134 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7gx62" Oct 30 21:48:43.825425 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.825411 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7gx62" Oct 30 21:48:43.902180 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.902149 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/556c8f6eb09b355e5322f215128e2cca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal\" (UID: \"556c8f6eb09b355e5322f215128e2cca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:48:43.902180 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.902163 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/556c8f6eb09b355e5322f215128e2cca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal\" (UID: \"556c8f6eb09b355e5322f215128e2cca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:48:43.902395 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.902202 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/774172d5b7cfcf0d4b41c09fc8c830a1-config\") pod \"kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal\" (UID: \"774172d5b7cfcf0d4b41c09fc8c830a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:48:43.902395 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.902225 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/556c8f6eb09b355e5322f215128e2cca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal\" (UID: \"556c8f6eb09b355e5322f215128e2cca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:48:43.902395 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.902267 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/556c8f6eb09b355e5322f215128e2cca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal\" (UID: \"556c8f6eb09b355e5322f215128e2cca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:48:43.902395 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.902287 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/774172d5b7cfcf0d4b41c09fc8c830a1-config\") pod \"kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal\" (UID: \"774172d5b7cfcf0d4b41c09fc8c830a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:48:43.908356 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.908337 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:48:43.923352 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:43.923328 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:48:43.923445 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:43.923355 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-7.us-west-2.compute.internal\": node \"ip-10-0-135-7.us-west-2.compute.internal\" not found" Oct 30 21:48:44.120986 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:44.120938 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:48:44.124882 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:44.124857 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:48:44.136932 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:44.136907 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-7.us-west-2.compute.internal\" not found" Oct 30 21:48:44.238983 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:44.238942 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-7.us-west-2.compute.internal\" not found" Oct 30 21:48:44.341328 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:44.341215 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-7.us-west-2.compute.internal\" not found" Oct 30 21:48:44.444023 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:44.443981 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-7.us-west-2.compute.internal\" not found" Oct 30 21:48:44.528993 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:44.528954 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 21:48:44.529595 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:44.529192 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:48:44.529595 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:44.529214 2578 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a8a1717a8546b4318bd8725a26f164b5-359068c9b1489aca.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-135-7.us-west-2.compute.internal.18e36d000daf7694\": read tcp 10.0.135.7:56078->52.15.155.115:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-135-7.us-west-2.compute.internal.18e36d000daf7694 default 4815 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-7.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-7.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-7.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:48:43 +0000 UTC,LastTimestamp:2026-10-30 21:48:43.77738794 +0000 UTC m=+0.597121486,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-7.us-west-2.compute.internal,}" Oct 30 21:48:44.544573 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:44.544532 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-7.us-west-2.compute.internal\" not found" Oct 30 21:48:44.646978 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:44.646902 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-7.us-west-2.compute.internal\" not found" Oct 30 21:48:44.747227 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:44.747191 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-7.us-west-2.compute.internal\" not found" Oct 30 21:48:44.831412 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:44.831362 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:43:44 +0000 UTC" deadline="2028-06-24 05:19:01.538307792 +0000 UTC" Oct 30 21:48:44.831412 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:44.831404 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14455h30m16.706906866s" Oct 30 21:48:44.847751 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:44.847716 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-7.us-west-2.compute.internal\" not found" Oct 30 21:48:44.869127 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:44.869097 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:48:44.872107 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:44.870686 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:48:44.872240 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:44.872140 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:48:44.872240 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:44.872181 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:48:44.948065 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:44.947974 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-7.us-west-2.compute.internal\" not found" Oct 30 21:48:45.048783 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:45.048746 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-7.us-west-2.compute.internal\" not found" Oct 30 21:48:45.149241 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:45.149197 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-7.us-west-2.compute.internal\" not found" Oct 30 21:48:45.211188 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.211108 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:48:45.250449 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:45.250404 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-7.us-west-2.compute.internal\" not found" Oct 30 21:48:45.261305 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.261278 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:48:45.300490 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.300454 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:48:45.362113 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:45.362076 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556c8f6eb09b355e5322f215128e2cca.slice/crio-a8f41036b33b5ee78d7094694eb5bbd534b732f7cc74ed189e023f1179486594 WatchSource:0}: Error finding container a8f41036b33b5ee78d7094694eb5bbd534b732f7cc74ed189e023f1179486594: Status 404 returned error can't find the container with id a8f41036b33b5ee78d7094694eb5bbd534b732f7cc74ed189e023f1179486594 Oct 30 21:48:45.362535 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:45.362513 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774172d5b7cfcf0d4b41c09fc8c830a1.slice/crio-807ca5626e858e6a12ee34390615264e14111f00424b2e4b7942a71865766972 WatchSource:0}: Error finding container 807ca5626e858e6a12ee34390615264e14111f00424b2e4b7942a71865766972: Status 404 returned error can't find the container with id 807ca5626e858e6a12ee34390615264e14111f00424b2e4b7942a71865766972 Oct 30 21:48:45.368645 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.368628 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:48:45.372748 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.372732 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:48:45.374701 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.374686 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:48:45.492894 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.492811 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:48:45.492894 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.492865 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:48:45.573703 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.573673 2578 apiserver.go:52] "Watching apiserver" Oct 30 21:48:45.601991 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.601955 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 21:48:45.608127 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.608104 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:48:45.683407 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.683343 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal" event={"ID":"556c8f6eb09b355e5322f215128e2cca","Type":"ContainerStarted","Data":"a8f41036b33b5ee78d7094694eb5bbd534b732f7cc74ed189e023f1179486594"} Oct 30 21:48:45.684454 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.684427 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal" event={"ID":"774172d5b7cfcf0d4b41c09fc8c830a1","Type":"ContainerStarted","Data":"807ca5626e858e6a12ee34390615264e14111f00424b2e4b7942a71865766972"} Oct 30 21:48:45.694326 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.694305 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:48:45.694651 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.694634 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-w2t6m","openshift-multus/multus-mmdsp","kube-system/global-pull-secret-syncer-lfq7z","openshift-network-diagnostics/network-check-target-6gzkm","openshift-ovn-kubernetes/ovnkube-node-vtgcr","kube-system/konnectivity-agent-929rz","kube-system/kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-ld4hs","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8","openshift-dns/node-resolver-74zqz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-69j9v","openshift-network-operator/iptables-alerter-gft82","openshift-image-registry/node-ca-x4655"] Oct 30 21:48:45.698691 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.698675 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:48:45.698775 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:45.698757 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69j9v" podUID="bd3b421a-9313-4f15-b81f-900717685fd6" Oct 30 21:48:45.700908 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.700885 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.703175 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.703155 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x4655" Oct 30 21:48:45.705516 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.705499 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:48:45.705616 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:45.705572 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gzkm" podUID="550e690a-605e-47a4-ba95-74bcd26e0cc3" Oct 30 21:48:45.708043 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.708022 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.710527 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.710508 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/13fd2c81-54ab-4afa-9afe-68094563bf08-multus-daemon-config\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.710604 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.710537 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-etc-kubernetes\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.710604 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.710557 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs\") pod \"network-metrics-daemon-69j9v\" (UID: \"bd3b421a-9313-4f15-b81f-900717685fd6\") " pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:48:45.710604 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.710563 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-929rz" Oct 30 21:48:45.710604 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.710573 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-multus-cni-dir\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.710604 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.710590 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-run-k8s-cni-cncf-io\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.710864 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.710607 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-hostroot\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.710864 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.710671 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2qx6\" (UniqueName: \"kubernetes.io/projected/bd3b421a-9313-4f15-b81f-900717685fd6-kube-api-access-j2qx6\") pod \"network-metrics-daemon-69j9v\" (UID: \"bd3b421a-9313-4f15-b81f-900717685fd6\") " pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:48:45.710864 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.710711 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-multus-socket-dir-parent\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.710864 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.710764 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-var-lib-kubelet\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.710864 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.710818 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-multus-conf-dir\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.711129 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.710888 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-cnibin\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.711129 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.710931 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-run-netns\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.711129 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.710991 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-var-lib-cni-multus\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.711129 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.711030 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-run-multus-certs\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.711129 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.711052 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc5dn\" (UniqueName: \"kubernetes.io/projected/13fd2c81-54ab-4afa-9afe-68094563bf08-kube-api-access-vc5dn\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.711129 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.711103 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-system-cni-dir\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.711418 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.711139 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13fd2c81-54ab-4afa-9afe-68094563bf08-cni-binary-copy\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.711418 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.711167 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-var-lib-cni-bin\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.713052 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.713036 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gft82" Oct 30 21:48:45.715647 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.715628 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.718791 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.718764 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:48:45.721346 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.721326 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:48:45.723806 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.723787 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-74zqz" Oct 30 21:48:45.726227 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.726204 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:48:45.726347 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:45.726268 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lfq7z" podUID="1d430f7e-00f1-4b9e-9427-3e9971cfff31" Oct 30 21:48:45.762541 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.762523 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-92jf8\"" Oct 30 21:48:45.763109 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.763093 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:48:45.763213 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.763199 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cmhxq\"" Oct 30 21:48:45.763451 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.763436 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:48:45.763511 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.763459 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:48:45.763511 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.763473 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:48:45.763617 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.763570 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:48:45.763617 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.763581 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:48:45.763687 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.763675 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:48:45.773362 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.773344 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:48:45.773362 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.773352 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:48:45.773520 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.773506 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:48:45.773807 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.773793 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:48:45.773880 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.773816 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:48:45.773915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.773884 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hm4gw\"" Oct 30 21:48:45.774203 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.774190 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:48:45.775940 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.775918 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:48:45.776039 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.776027 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:48:45.776247 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.776234 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9mttd\"" Oct 30 21:48:45.776303 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.776243 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nwhdt\"" Oct 30 21:48:45.776303 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.776291 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:48:45.776410 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.776314 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:48:45.776474 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.776461 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:48:45.776561 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.776550 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:48:45.776742 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.776732 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:48:45.802574 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.802548 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:48:45.811656 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.811632 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-systemd-units\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.811743 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.811661 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7aa1b80f-115e-4708-bc98-88d13ad66860-ovnkube-config\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.811743 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.811682 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ceedf44e-88c1-49cf-901e-75558703c361-serviceca\") pod \"node-ca-x4655\" (UID: \"ceedf44e-88c1-49cf-901e-75558703c361\") " pod="openshift-image-registry/node-ca-x4655" Oct 30 21:48:45.811743 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.811699 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk7l4\" (UniqueName: \"kubernetes.io/projected/550e690a-605e-47a4-ba95-74bcd26e0cc3-kube-api-access-kk7l4\") pod \"network-check-target-6gzkm\" (UID: \"550e690a-605e-47a4-ba95-74bcd26e0cc3\") " pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:48:45.811743 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.811720 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1d430f7e-00f1-4b9e-9427-3e9971cfff31-kubelet-config\") pod \"global-pull-secret-syncer-lfq7z\" (UID: \"1d430f7e-00f1-4b9e-9427-3e9971cfff31\") " pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:48:45.811921 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.811783 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1d430f7e-00f1-4b9e-9427-3e9971cfff31-dbus\") pod \"global-pull-secret-syncer-lfq7z\" (UID: \"1d430f7e-00f1-4b9e-9427-3e9971cfff31\") " pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:48:45.811921 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.811826 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-kubernetes\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.811921 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.811851 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-multus-cni-dir\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.812022 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.811964 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-run-k8s-cni-cncf-io\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.812022 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.811987 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-kubelet\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.812022 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812003 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-sysctl-d\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.812133 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812021 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-sysctl-conf\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.812133 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812032 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-multus-cni-dir\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.812133 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812046 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-lib-modules\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.812133 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812071 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-run-ovn\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.812133 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812095 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-node-log\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.812133 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812107 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-run-k8s-cni-cncf-io\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.812133 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812122 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-cni-netd\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.812422 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812162 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7aa1b80f-115e-4708-bc98-88d13ad66860-ovnkube-script-lib\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.812422 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812185 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffa64e38-3f1b-49af-913f-139addefa359-cnibin\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:48:45.812422 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812216 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-modprobe-d\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.812422 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812240 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-systemd\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.812422 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812297 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-tuned\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.812422 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812336 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/64086350-55f6-4da5-a871-3f23c1755371-agent-certs\") pod \"konnectivity-agent-929rz\" (UID: \"64086350-55f6-4da5-a871-3f23c1755371\") " pod="kube-system/konnectivity-agent-929rz" Oct 30 21:48:45.812422 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812355 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/64086350-55f6-4da5-a871-3f23c1755371-konnectivity-ca\") pod \"konnectivity-agent-929rz\" (UID: \"64086350-55f6-4da5-a871-3f23c1755371\") " pod="kube-system/konnectivity-agent-929rz" Oct 30 21:48:45.812422 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812384 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9161cdbd-c44f-4a8c-b918-107fc7001f3a-iptables-alerter-script\") pod \"iptables-alerter-gft82\" (UID: \"9161cdbd-c44f-4a8c-b918-107fc7001f3a\") " pod="openshift-network-operator/iptables-alerter-gft82" Oct 30 21:48:45.812422 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812414 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-sys-fs\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:48:45.812755 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812433 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-etc-kubernetes\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.812755 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812460 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs\") pod \"network-metrics-daemon-69j9v\" (UID: \"bd3b421a-9313-4f15-b81f-900717685fd6\") " pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:48:45.812755 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812485 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-etc-kubernetes\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.812755 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812516 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7aa1b80f-115e-4708-bc98-88d13ad66860-env-overrides\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.812755 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812537 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffa64e38-3f1b-49af-913f-139addefa359-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:48:45.812755 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:45.812617 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:48:45.812755 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812647 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6k4l\" (UniqueName: \"kubernetes.io/projected/a32b60f2-3000-4fe8-a928-955de8c76c82-kube-api-access-r6k4l\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.812755 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:45.812703 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs podName:bd3b421a-9313-4f15-b81f-900717685fd6 nodeName:}" failed. No retries permitted until 2026-10-30 21:48:46.31266809 +0000 UTC m=+3.132401646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs") pod "network-metrics-daemon-69j9v" (UID: "bd3b421a-9313-4f15-b81f-900717685fd6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:48:45.812755 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812741 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q7c2\" (UniqueName: \"kubernetes.io/projected/1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4-kube-api-access-7q7c2\") pod \"node-resolver-74zqz\" (UID: \"1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4\") " pod="openshift-dns/node-resolver-74zqz" Oct 30 21:48:45.813074 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812763 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2qx6\" (UniqueName: \"kubernetes.io/projected/bd3b421a-9313-4f15-b81f-900717685fd6-kube-api-access-j2qx6\") pod \"network-metrics-daemon-69j9v\" (UID: \"bd3b421a-9313-4f15-b81f-900717685fd6\") " pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:48:45.813074 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812779 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-run-systemd\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.813074 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812802 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7aa1b80f-115e-4708-bc98-88d13ad66860-ovn-node-metrics-cert\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.813074 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812816 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hz67\" (UniqueName: \"kubernetes.io/projected/9161cdbd-c44f-4a8c-b918-107fc7001f3a-kube-api-access-9hz67\") pod \"iptables-alerter-gft82\" (UID: \"9161cdbd-c44f-4a8c-b918-107fc7001f3a\") " pod="openshift-network-operator/iptables-alerter-gft82" Oct 30 21:48:45.813074 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812851 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw4xv\" (UniqueName: \"kubernetes.io/projected/b2f2c486-d661-4d59-981c-0b95a1f66fb4-kube-api-access-sw4xv\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:48:45.813074 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812873 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret\") pod \"global-pull-secret-syncer-lfq7z\" (UID: \"1d430f7e-00f1-4b9e-9427-3e9971cfff31\") " pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:48:45.813074 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812894 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-cnibin\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.813074 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812927 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-var-lib-cni-multus\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.813074 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812948 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-var-lib-cni-bin\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.813074 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812966 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-slash\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.813074 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.812988 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffa64e38-3f1b-49af-913f-139addefa359-cni-binary-copy\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:48:45.813074 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813012 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-var-lib-cni-bin\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.813074 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813036 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-var-lib-cni-multus\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.813478 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813091 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-cnibin\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.813478 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813116 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fgkx\" (UniqueName: \"kubernetes.io/projected/ffa64e38-3f1b-49af-913f-139addefa359-kube-api-access-5fgkx\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:48:45.813478 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813183 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8pl2\" (UniqueName: \"kubernetes.io/projected/ceedf44e-88c1-49cf-901e-75558703c361-kube-api-access-s8pl2\") pod \"node-ca-x4655\" (UID: \"ceedf44e-88c1-49cf-901e-75558703c361\") " pod="openshift-image-registry/node-ca-x4655" Oct 30 21:48:45.813478 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813214 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-sysconfig\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.813478 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813232 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-run\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.813478 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813278 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-hostroot\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.813478 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813304 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-run-openvswitch\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.813478 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813331 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-hostroot\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.813478 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813352 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-cni-bin\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.813478 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813371 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.813478 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813399 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ffa64e38-3f1b-49af-913f-139addefa359-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:48:45.813478 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813427 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-socket-dir\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:48:45.813478 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813446 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-registration-dir\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:48:45.813478 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813469 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-device-dir\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:48:45.814042 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813500 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-multus-socket-dir-parent\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.814042 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813522 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-multus-conf-dir\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.814042 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813549 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-etc-selinux\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:48:45.814042 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813583 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-multus-conf-dir\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.814042 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813585 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-multus-socket-dir-parent\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.814042 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813597 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-run-netns\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.814042 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813630 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc5dn\" (UniqueName: \"kubernetes.io/projected/13fd2c81-54ab-4afa-9afe-68094563bf08-kube-api-access-vc5dn\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.814042 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813649 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-run-netns\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.814042 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813669 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-etc-openvswitch\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.814042 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813675 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-run-netns\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.814042 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813693 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-var-lib-kubelet\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.814042 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813723 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:48:45.814042 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813739 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:48:45.814042 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813763 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4-hosts-file\") pod \"node-resolver-74zqz\" (UID: \"1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4\") " pod="openshift-dns/node-resolver-74zqz" Oct 30 21:48:45.814042 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813779 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-h8zdp" Oct 30 21:48:45.814042 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813794 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-system-cni-dir\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.814042 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813835 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13fd2c81-54ab-4afa-9afe-68094563bf08-cni-binary-copy\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.814042 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813877 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/13fd2c81-54ab-4afa-9afe-68094563bf08-multus-daemon-config\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.815044 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813899 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-log-socket\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.815044 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813917 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-run-ovn-kubernetes\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.815044 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813933 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-sys\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.815044 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813948 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-host\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.815044 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813940 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-system-cni-dir\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.815044 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.813970 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9161cdbd-c44f-4a8c-b918-107fc7001f3a-host-slash\") pod \"iptables-alerter-gft82\" (UID: \"9161cdbd-c44f-4a8c-b918-107fc7001f3a\") " pod="openshift-network-operator/iptables-alerter-gft82" Oct 30 21:48:45.815044 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.814010 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a32b60f2-3000-4fe8-a928-955de8c76c82-tmp\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.815044 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.814046 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-var-lib-kubelet\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.815044 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.814078 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-var-lib-openvswitch\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.815044 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.814088 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-var-lib-kubelet\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.815044 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.814115 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvxrm\" (UniqueName: \"kubernetes.io/projected/7aa1b80f-115e-4708-bc98-88d13ad66860-kube-api-access-qvxrm\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.815044 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.814142 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffa64e38-3f1b-49af-913f-139addefa359-system-cni-dir\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:48:45.815044 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.814174 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ffa64e38-3f1b-49af-913f-139addefa359-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:48:45.815677 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.815169 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13fd2c81-54ab-4afa-9afe-68094563bf08-cni-binary-copy\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.815677 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.815187 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ceedf44e-88c1-49cf-901e-75558703c361-host\") pod \"node-ca-x4655\" (UID: \"ceedf44e-88c1-49cf-901e-75558703c361\") " pod="openshift-image-registry/node-ca-x4655" Oct 30 21:48:45.815677 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.815225 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4-tmp-dir\") pod \"node-resolver-74zqz\" (UID: \"1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4\") " pod="openshift-dns/node-resolver-74zqz" Oct 30 21:48:45.815677 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.815411 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-run-multus-certs\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.815677 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.815427 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/13fd2c81-54ab-4afa-9afe-68094563bf08-multus-daemon-config\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.815677 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.815491 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-run-multus-certs\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.872837 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.872805 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gbc4r\"" Oct 30 21:48:45.873021 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.872942 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:48:45.873021 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.872981 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:48:45.873021 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.873011 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:48:45.873177 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.873044 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7d2ft\"" Oct 30 21:48:45.873177 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.873068 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dvh7z\"" Oct 30 21:48:45.873177 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.873077 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:48:45.873177 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.873070 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6gwn9\"" Oct 30 21:48:45.873177 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.873075 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:48:45.873177 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.873107 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:48:45.873177 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.873141 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:48:45.879823 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.879805 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:48:45.883459 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.883434 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc5dn\" (UniqueName: \"kubernetes.io/projected/13fd2c81-54ab-4afa-9afe-68094563bf08-kube-api-access-vc5dn\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:48:45.883567 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.883434 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2qx6\" (UniqueName: \"kubernetes.io/projected/bd3b421a-9313-4f15-b81f-900717685fd6-kube-api-access-j2qx6\") pod \"network-metrics-daemon-69j9v\" (UID: \"bd3b421a-9313-4f15-b81f-900717685fd6\") " pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:48:45.916736 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.916703 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffa64e38-3f1b-49af-913f-139addefa359-cni-binary-copy\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:48:45.916847 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.916740 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fgkx\" (UniqueName: \"kubernetes.io/projected/ffa64e38-3f1b-49af-913f-139addefa359-kube-api-access-5fgkx\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:48:45.916847 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.916760 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8pl2\" (UniqueName: \"kubernetes.io/projected/ceedf44e-88c1-49cf-901e-75558703c361-kube-api-access-s8pl2\") pod \"node-ca-x4655\" (UID: \"ceedf44e-88c1-49cf-901e-75558703c361\") " pod="openshift-image-registry/node-ca-x4655" Oct 30 21:48:45.916847 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.916778 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-sysconfig\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.916847 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.916798 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-run\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.916847 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.916824 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-run-openvswitch\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.916847 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.916846 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-cni-bin\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.917138 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.916890 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-sysconfig\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.917138 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.916899 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-cni-bin\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.917138 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.916905 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-run-openvswitch\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.917138 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.916947 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-run\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.917138 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.916983 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.917138 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917044 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ffa64e38-3f1b-49af-913f-139addefa359-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:48:45.917138 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917079 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-socket-dir\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:48:45.917138 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917106 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.917138 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917129 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-registration-dir\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:48:45.917513 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917170 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-device-dir\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:48:45.917513 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917214 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-registration-dir\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:48:45.917513 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917227 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-etc-selinux\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:48:45.917513 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917256 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-run-netns\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.917513 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917300 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-etc-openvswitch\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.917513 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917315 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-etc-selinux\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:48:45.917513 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917275 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-device-dir\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:48:45.917513 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917326 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-var-lib-kubelet\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.917513 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917368 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-etc-openvswitch\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.917513 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917381 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-var-lib-kubelet\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.917513 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917324 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-socket-dir\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:48:45.917513 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917384 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-run-netns\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.917513 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917398 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:48:45.917513 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917405 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffa64e38-3f1b-49af-913f-139addefa359-cni-binary-copy\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:48:45.917513 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917373 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:48:45.917513 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917463 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4-hosts-file\") pod \"node-resolver-74zqz\" (UID: \"1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4\") " pod="openshift-dns/node-resolver-74zqz" Oct 30 21:48:45.917513 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917495 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-log-socket\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.918246 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917524 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-run-ovn-kubernetes\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.918246 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917560 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-sys\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.918246 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917568 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-log-socket\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.918246 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917582 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-host\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.918246 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917589 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4-hosts-file\") pod \"node-resolver-74zqz\" (UID: \"1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4\") " pod="openshift-dns/node-resolver-74zqz" Oct 30 21:48:45.918246 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917599 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-run-ovn-kubernetes\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.918246 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917608 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9161cdbd-c44f-4a8c-b918-107fc7001f3a-host-slash\") pod \"iptables-alerter-gft82\" (UID: \"9161cdbd-c44f-4a8c-b918-107fc7001f3a\") " pod="openshift-network-operator/iptables-alerter-gft82" Oct 30 21:48:45.918246 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917638 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-sys\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.918246 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917641 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a32b60f2-3000-4fe8-a928-955de8c76c82-tmp\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.918246 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917650 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9161cdbd-c44f-4a8c-b918-107fc7001f3a-host-slash\") pod \"iptables-alerter-gft82\" (UID: \"9161cdbd-c44f-4a8c-b918-107fc7001f3a\") " pod="openshift-network-operator/iptables-alerter-gft82" Oct 30 21:48:45.918246 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917650 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-host\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.918246 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917662 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ffa64e38-3f1b-49af-913f-139addefa359-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:48:45.918246 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917668 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-var-lib-openvswitch\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.918246 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917704 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-var-lib-openvswitch\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.918246 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917719 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvxrm\" (UniqueName: \"kubernetes.io/projected/7aa1b80f-115e-4708-bc98-88d13ad66860-kube-api-access-qvxrm\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.918246 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917744 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffa64e38-3f1b-49af-913f-139addefa359-system-cni-dir\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:48:45.918246 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917761 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ffa64e38-3f1b-49af-913f-139addefa359-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:48:45.919103 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917778 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ceedf44e-88c1-49cf-901e-75558703c361-host\") pod \"node-ca-x4655\" (UID: \"ceedf44e-88c1-49cf-901e-75558703c361\") " pod="openshift-image-registry/node-ca-x4655" Oct 30 21:48:45.919103 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917803 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffa64e38-3f1b-49af-913f-139addefa359-system-cni-dir\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:48:45.919103 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917806 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4-tmp-dir\") pod \"node-resolver-74zqz\" (UID: \"1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4\") " pod="openshift-dns/node-resolver-74zqz" Oct 30 21:48:45.919103 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917834 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-systemd-units\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.919103 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917858 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7aa1b80f-115e-4708-bc98-88d13ad66860-ovnkube-config\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.919103 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917882 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ceedf44e-88c1-49cf-901e-75558703c361-serviceca\") pod \"node-ca-x4655\" (UID: \"ceedf44e-88c1-49cf-901e-75558703c361\") " pod="openshift-image-registry/node-ca-x4655" Oct 30 21:48:45.919103 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917908 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk7l4\" (UniqueName: \"kubernetes.io/projected/550e690a-605e-47a4-ba95-74bcd26e0cc3-kube-api-access-kk7l4\") pod \"network-check-target-6gzkm\" (UID: \"550e690a-605e-47a4-ba95-74bcd26e0cc3\") " pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:48:45.919103 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917925 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-systemd-units\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.919103 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917935 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1d430f7e-00f1-4b9e-9427-3e9971cfff31-kubelet-config\") pod \"global-pull-secret-syncer-lfq7z\" (UID: \"1d430f7e-00f1-4b9e-9427-3e9971cfff31\") " pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:48:45.919103 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917962 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1d430f7e-00f1-4b9e-9427-3e9971cfff31-dbus\") pod \"global-pull-secret-syncer-lfq7z\" (UID: \"1d430f7e-00f1-4b9e-9427-3e9971cfff31\") " pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:48:45.919103 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.917986 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-kubernetes\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.919103 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-kubelet\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.919103 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918042 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-sysctl-d\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.919103 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918065 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-sysctl-conf\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.919103 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918057 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1d430f7e-00f1-4b9e-9427-3e9971cfff31-kubelet-config\") pod \"global-pull-secret-syncer-lfq7z\" (UID: \"1d430f7e-00f1-4b9e-9427-3e9971cfff31\") " pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:48:45.919103 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918137 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-kubelet\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.919103 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918142 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1d430f7e-00f1-4b9e-9427-3e9971cfff31-dbus\") pod \"global-pull-secret-syncer-lfq7z\" (UID: \"1d430f7e-00f1-4b9e-9427-3e9971cfff31\") " pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:48:45.919103 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918196 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-kubernetes\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.919915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918257 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-sysctl-d\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.919915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918302 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ceedf44e-88c1-49cf-901e-75558703c361-host\") pod \"node-ca-x4655\" (UID: \"ceedf44e-88c1-49cf-901e-75558703c361\") " pod="openshift-image-registry/node-ca-x4655" Oct 30 21:48:45.919915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918306 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-lib-modules\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.919915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918322 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-sysctl-conf\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.919915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918341 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-run-ovn\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.919915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918370 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-node-log\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.919915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918393 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-cni-netd\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.919915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918411 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-lib-modules\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.919915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918417 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7aa1b80f-115e-4708-bc98-88d13ad66860-ovnkube-script-lib\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.919915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918423 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ffa64e38-3f1b-49af-913f-139addefa359-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:48:45.919915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918444 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffa64e38-3f1b-49af-913f-139addefa359-cnibin\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:48:45.919915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918448 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-node-log\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.919915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918464 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ceedf44e-88c1-49cf-901e-75558703c361-serviceca\") pod \"node-ca-x4655\" (UID: \"ceedf44e-88c1-49cf-901e-75558703c361\") " pod="openshift-image-registry/node-ca-x4655" Oct 30 21:48:45.919915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918470 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-modprobe-d\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.919915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918494 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-run-ovn\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.919915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918497 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffa64e38-3f1b-49af-913f-139addefa359-cnibin\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:48:45.919915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918513 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-systemd\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.919915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918538 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-cni-netd\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.920591 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918541 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-tuned\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.920591 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918568 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-modprobe-d\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.920591 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918594 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/64086350-55f6-4da5-a871-3f23c1755371-agent-certs\") pod \"konnectivity-agent-929rz\" (UID: \"64086350-55f6-4da5-a871-3f23c1755371\") " pod="kube-system/konnectivity-agent-929rz" Oct 30 21:48:45.920591 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918622 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/64086350-55f6-4da5-a871-3f23c1755371-konnectivity-ca\") pod \"konnectivity-agent-929rz\" (UID: \"64086350-55f6-4da5-a871-3f23c1755371\") " pod="kube-system/konnectivity-agent-929rz" Oct 30 21:48:45.920591 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918639 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-systemd\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.920591 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918648 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9161cdbd-c44f-4a8c-b918-107fc7001f3a-iptables-alerter-script\") pod \"iptables-alerter-gft82\" (UID: \"9161cdbd-c44f-4a8c-b918-107fc7001f3a\") " pod="openshift-network-operator/iptables-alerter-gft82" Oct 30 21:48:45.920591 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918675 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-sys-fs\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:48:45.920591 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918703 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4-tmp-dir\") pod \"node-resolver-74zqz\" (UID: \"1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4\") " pod="openshift-dns/node-resolver-74zqz" Oct 30 21:48:45.920591 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918717 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7aa1b80f-115e-4708-bc98-88d13ad66860-env-overrides\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.920591 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918732 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7aa1b80f-115e-4708-bc98-88d13ad66860-ovnkube-config\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.920591 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918744 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffa64e38-3f1b-49af-913f-139addefa359-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:48:45.920591 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918772 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6k4l\" (UniqueName: \"kubernetes.io/projected/a32b60f2-3000-4fe8-a928-955de8c76c82-kube-api-access-r6k4l\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.920591 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918798 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q7c2\" (UniqueName: \"kubernetes.io/projected/1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4-kube-api-access-7q7c2\") pod \"node-resolver-74zqz\" (UID: \"1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4\") " pod="openshift-dns/node-resolver-74zqz" Oct 30 21:48:45.920591 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918828 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-run-systemd\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.920591 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918865 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7aa1b80f-115e-4708-bc98-88d13ad66860-ovn-node-metrics-cert\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.920591 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hz67\" (UniqueName: \"kubernetes.io/projected/9161cdbd-c44f-4a8c-b918-107fc7001f3a-kube-api-access-9hz67\") pod \"iptables-alerter-gft82\" (UID: \"9161cdbd-c44f-4a8c-b918-107fc7001f3a\") " pod="openshift-network-operator/iptables-alerter-gft82" Oct 30 21:48:45.920591 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918918 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw4xv\" (UniqueName: \"kubernetes.io/projected/b2f2c486-d661-4d59-981c-0b95a1f66fb4-kube-api-access-sw4xv\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:48:45.921382 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918944 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret\") pod \"global-pull-secret-syncer-lfq7z\" (UID: \"1d430f7e-00f1-4b9e-9427-3e9971cfff31\") " pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:48:45.921382 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.918976 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-slash\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.921382 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.919009 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7aa1b80f-115e-4708-bc98-88d13ad66860-ovnkube-script-lib\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.921382 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.919049 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-slash\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.921382 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.919075 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-sys-fs\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:48:45.921382 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.919091 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-run-systemd\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.921382 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:45.919443 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:48:45.921382 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:45.919505 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret podName:1d430f7e-00f1-4b9e-9427-3e9971cfff31 nodeName:}" failed. No retries permitted until 2026-10-30 21:48:46.419491542 +0000 UTC m=+3.239225081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret") pod "global-pull-secret-syncer-lfq7z" (UID: "1d430f7e-00f1-4b9e-9427-3e9971cfff31") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:48:45.921382 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.919968 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffa64e38-3f1b-49af-913f-139addefa359-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:48:45.921382 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.920958 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9161cdbd-c44f-4a8c-b918-107fc7001f3a-iptables-alerter-script\") pod \"iptables-alerter-gft82\" (UID: \"9161cdbd-c44f-4a8c-b918-107fc7001f3a\") " pod="openshift-network-operator/iptables-alerter-gft82" Oct 30 21:48:45.921382 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.921005 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/64086350-55f6-4da5-a871-3f23c1755371-konnectivity-ca\") pod \"konnectivity-agent-929rz\" (UID: \"64086350-55f6-4da5-a871-3f23c1755371\") " pod="kube-system/konnectivity-agent-929rz" Oct 30 21:48:45.921382 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.921008 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7aa1b80f-115e-4708-bc98-88d13ad66860-env-overrides\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.921382 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.921082 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a32b60f2-3000-4fe8-a928-955de8c76c82-tmp\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.921382 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.921137 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-tuned\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.921847 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.921445 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/64086350-55f6-4da5-a871-3f23c1755371-agent-certs\") pod \"konnectivity-agent-929rz\" (UID: \"64086350-55f6-4da5-a871-3f23c1755371\") " pod="kube-system/konnectivity-agent-929rz" Oct 30 21:48:45.922018 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.922000 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7aa1b80f-115e-4708-bc98-88d13ad66860-ovn-node-metrics-cert\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.983161 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:45.983127 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:48:45.983161 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:45.983164 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:48:45.983385 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:45.983180 2578 projected.go:196] Error preparing data for projected volume kube-api-access-kk7l4 for pod openshift-network-diagnostics/network-check-target-6gzkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:48:45.983385 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:45.983291 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/550e690a-605e-47a4-ba95-74bcd26e0cc3-kube-api-access-kk7l4 podName:550e690a-605e-47a4-ba95-74bcd26e0cc3 nodeName:}" failed. No retries permitted until 2026-10-30 21:48:46.48324625 +0000 UTC m=+3.302979807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kk7l4" (UniqueName: "kubernetes.io/projected/550e690a-605e-47a4-ba95-74bcd26e0cc3-kube-api-access-kk7l4") pod "network-check-target-6gzkm" (UID: "550e690a-605e-47a4-ba95-74bcd26e0cc3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:48:45.984984 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.984955 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fgkx\" (UniqueName: \"kubernetes.io/projected/ffa64e38-3f1b-49af-913f-139addefa359-kube-api-access-5fgkx\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:48:45.985117 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.985060 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8pl2\" (UniqueName: \"kubernetes.io/projected/ceedf44e-88c1-49cf-901e-75558703c361-kube-api-access-s8pl2\") pod \"node-ca-x4655\" (UID: \"ceedf44e-88c1-49cf-901e-75558703c361\") " pod="openshift-image-registry/node-ca-x4655" Oct 30 21:48:45.985436 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.985414 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvxrm\" (UniqueName: \"kubernetes.io/projected/7aa1b80f-115e-4708-bc98-88d13ad66860-kube-api-access-qvxrm\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:45.986334 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.986310 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q7c2\" (UniqueName: \"kubernetes.io/projected/1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4-kube-api-access-7q7c2\") pod \"node-resolver-74zqz\" (UID: \"1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4\") " pod="openshift-dns/node-resolver-74zqz" Oct 30 21:48:45.987082 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.987064 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6k4l\" (UniqueName: \"kubernetes.io/projected/a32b60f2-3000-4fe8-a928-955de8c76c82-kube-api-access-r6k4l\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:45.990622 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.990597 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hz67\" (UniqueName: \"kubernetes.io/projected/9161cdbd-c44f-4a8c-b918-107fc7001f3a-kube-api-access-9hz67\") pod \"iptables-alerter-gft82\" (UID: \"9161cdbd-c44f-4a8c-b918-107fc7001f3a\") " pod="openshift-network-operator/iptables-alerter-gft82" Oct 30 21:48:45.992951 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:45.992927 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw4xv\" (UniqueName: \"kubernetes.io/projected/b2f2c486-d661-4d59-981c-0b95a1f66fb4-kube-api-access-sw4xv\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:48:46.014343 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:46.014247 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mmdsp" Oct 30 21:48:46.023362 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:46.023339 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x4655" Oct 30 21:48:46.025166 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:46.025137 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13fd2c81_54ab_4afa_9afe_68094563bf08.slice/crio-ba241dfb9dc382019190ceb8b1d6fdeddcc1972e09a124e0be2b603a4e0d2cf5 WatchSource:0}: Error finding container ba241dfb9dc382019190ceb8b1d6fdeddcc1972e09a124e0be2b603a4e0d2cf5: Status 404 returned error can't find the container with id ba241dfb9dc382019190ceb8b1d6fdeddcc1972e09a124e0be2b603a4e0d2cf5 Oct 30 21:48:46.032047 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:46.032024 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceedf44e_88c1_49cf_901e_75558703c361.slice/crio-163fe39a07496c2c1b0c6d137e35ddfe934972beac5729c3c172d8ceeb97c2d3 WatchSource:0}: Error finding container 163fe39a07496c2c1b0c6d137e35ddfe934972beac5729c3c172d8ceeb97c2d3: Status 404 returned error can't find the container with id 163fe39a07496c2c1b0c6d137e35ddfe934972beac5729c3c172d8ceeb97c2d3 Oct 30 21:48:46.042450 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:46.042426 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gft82" Oct 30 21:48:46.042561 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:46.042535 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-929rz" Oct 30 21:48:46.048454 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:46.048431 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:48:46.050720 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:46.050702 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:48:46.053760 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:46.053677 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9161cdbd_c44f_4a8c_b918_107fc7001f3a.slice/crio-372b376c17a9437a2aaed178dc1027b6d2f6bd6e7a3642656f226e4ab82e94b3 WatchSource:0}: Error finding container 372b376c17a9437a2aaed178dc1027b6d2f6bd6e7a3642656f226e4ab82e94b3: Status 404 returned error can't find the container with id 372b376c17a9437a2aaed178dc1027b6d2f6bd6e7a3642656f226e4ab82e94b3 Oct 30 21:48:46.057630 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:46.057610 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:48:46.062608 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:46.062584 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64086350_55f6_4da5_a871_3f23c1755371.slice/crio-a3983aaf8de073607eb578a07952b50edababd4cf514ebe150ae01940d0fbe8c WatchSource:0}: Error finding container a3983aaf8de073607eb578a07952b50edababd4cf514ebe150ae01940d0fbe8c: Status 404 returned error can't find the container with id a3983aaf8de073607eb578a07952b50edababd4cf514ebe150ae01940d0fbe8c Oct 30 21:48:46.066081 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:46.066055 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda32b60f2_3000_4fe8_a928_955de8c76c82.slice/crio-3bb6e87848390d29b46150c30380d7b64cff8b82bf51f0808b3951ecd62c4b98 WatchSource:0}: Error finding container 3bb6e87848390d29b46150c30380d7b64cff8b82bf51f0808b3951ecd62c4b98: Status 404 returned error can't find the container with id 3bb6e87848390d29b46150c30380d7b64cff8b82bf51f0808b3951ecd62c4b98 Oct 30 21:48:46.068096 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:46.068074 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-74zqz" Oct 30 21:48:46.068242 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:46.068216 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f2c486_d661_4d59_981c_0b95a1f66fb4.slice/crio-2aca39e96a0a5359c3c6f3557b7fab773e71a0f6396b23c99359d1f48424db5d WatchSource:0}: Error finding container 2aca39e96a0a5359c3c6f3557b7fab773e71a0f6396b23c99359d1f48424db5d: Status 404 returned error can't find the container with id 2aca39e96a0a5359c3c6f3557b7fab773e71a0f6396b23c99359d1f48424db5d Oct 30 21:48:46.075585 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:46.075565 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:48:46.078153 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:46.078129 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a96e3f0_b669_4a6b_a7fe_a9accbfc73b4.slice/crio-928ed5956456ef15f9e219522ca60366bbf675bbaca85d60cb4ed63fd76dbbfa WatchSource:0}: Error finding container 928ed5956456ef15f9e219522ca60366bbf675bbaca85d60cb4ed63fd76dbbfa: Status 404 returned error can't find the container with id 928ed5956456ef15f9e219522ca60366bbf675bbaca85d60cb4ed63fd76dbbfa Oct 30 21:48:46.084295 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:48:46.084270 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa64e38_3f1b_49af_913f_139addefa359.slice/crio-7e9c3b7469df82e4470afca5fe1a589a7407545d3785a7c0026ed27ecc696413 WatchSource:0}: Error finding container 7e9c3b7469df82e4470afca5fe1a589a7407545d3785a7c0026ed27ecc696413: Status 404 returned error can't find the container with id 7e9c3b7469df82e4470afca5fe1a589a7407545d3785a7c0026ed27ecc696413 Oct 30 21:48:46.322315 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:46.322193 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs\") pod \"network-metrics-daemon-69j9v\" (UID: \"bd3b421a-9313-4f15-b81f-900717685fd6\") " pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:48:46.322462 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:46.322366 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:48:46.322462 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:46.322426 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs podName:bd3b421a-9313-4f15-b81f-900717685fd6 nodeName:}" failed. No retries permitted until 2026-10-30 21:48:47.322413136 +0000 UTC m=+4.142146685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs") pod "network-metrics-daemon-69j9v" (UID: "bd3b421a-9313-4f15-b81f-900717685fd6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:48:46.423815 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:46.423720 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret\") pod \"global-pull-secret-syncer-lfq7z\" (UID: \"1d430f7e-00f1-4b9e-9427-3e9971cfff31\") " pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:48:46.423992 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:46.423826 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:48:46.423992 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:46.423915 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret podName:1d430f7e-00f1-4b9e-9427-3e9971cfff31 nodeName:}" failed. No retries permitted until 2026-10-30 21:48:47.423898846 +0000 UTC m=+4.243632385 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret") pod "global-pull-secret-syncer-lfq7z" (UID: "1d430f7e-00f1-4b9e-9427-3e9971cfff31") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:48:46.525116 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:46.525077 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk7l4\" (UniqueName: \"kubernetes.io/projected/550e690a-605e-47a4-ba95-74bcd26e0cc3-kube-api-access-kk7l4\") pod \"network-check-target-6gzkm\" (UID: \"550e690a-605e-47a4-ba95-74bcd26e0cc3\") " pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:48:46.525318 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:46.525285 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:48:46.525318 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:46.525308 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:48:46.525318 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:46.525320 2578 projected.go:196] Error preparing data for projected volume kube-api-access-kk7l4 for pod openshift-network-diagnostics/network-check-target-6gzkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:48:46.525442 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:46.525382 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/550e690a-605e-47a4-ba95-74bcd26e0cc3-kube-api-access-kk7l4 podName:550e690a-605e-47a4-ba95-74bcd26e0cc3 nodeName:}" failed. No retries permitted until 2026-10-30 21:48:47.525362158 +0000 UTC m=+4.345095711 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kk7l4" (UniqueName: "kubernetes.io/projected/550e690a-605e-47a4-ba95-74bcd26e0cc3-kube-api-access-kk7l4") pod "network-check-target-6gzkm" (UID: "550e690a-605e-47a4-ba95-74bcd26e0cc3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:48:46.688737 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:46.688635 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" event={"ID":"b2f2c486-d661-4d59-981c-0b95a1f66fb4","Type":"ContainerStarted","Data":"2aca39e96a0a5359c3c6f3557b7fab773e71a0f6396b23c99359d1f48424db5d"} Oct 30 21:48:46.690224 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:46.690199 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" event={"ID":"a32b60f2-3000-4fe8-a928-955de8c76c82","Type":"ContainerStarted","Data":"3bb6e87848390d29b46150c30380d7b64cff8b82bf51f0808b3951ecd62c4b98"} Oct 30 21:48:46.692349 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:46.692320 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-929rz" event={"ID":"64086350-55f6-4da5-a871-3f23c1755371","Type":"ContainerStarted","Data":"a3983aaf8de073607eb578a07952b50edababd4cf514ebe150ae01940d0fbe8c"} Oct 30 21:48:46.693611 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:46.693580 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x4655" event={"ID":"ceedf44e-88c1-49cf-901e-75558703c361","Type":"ContainerStarted","Data":"163fe39a07496c2c1b0c6d137e35ddfe934972beac5729c3c172d8ceeb97c2d3"} Oct 30 21:48:46.695419 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:46.695390 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mmdsp" event={"ID":"13fd2c81-54ab-4afa-9afe-68094563bf08","Type":"ContainerStarted","Data":"ba241dfb9dc382019190ceb8b1d6fdeddcc1972e09a124e0be2b603a4e0d2cf5"} Oct 30 21:48:46.697628 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:46.697600 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" event={"ID":"7aa1b80f-115e-4708-bc98-88d13ad66860","Type":"ContainerStarted","Data":"c65f9e423b79ea4d8c7c6522536eb48c1550c1bb189aad40fd355f8cd78429ef"} Oct 30 21:48:46.699195 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:46.699162 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gft82" event={"ID":"9161cdbd-c44f-4a8c-b918-107fc7001f3a","Type":"ContainerStarted","Data":"372b376c17a9437a2aaed178dc1027b6d2f6bd6e7a3642656f226e4ab82e94b3"} Oct 30 21:48:46.700559 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:46.700529 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2t6m" event={"ID":"ffa64e38-3f1b-49af-913f-139addefa359","Type":"ContainerStarted","Data":"7e9c3b7469df82e4470afca5fe1a589a7407545d3785a7c0026ed27ecc696413"} Oct 30 21:48:46.701845 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:46.701817 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-74zqz" event={"ID":"1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4","Type":"ContainerStarted","Data":"928ed5956456ef15f9e219522ca60366bbf675bbaca85d60cb4ed63fd76dbbfa"} Oct 30 21:48:46.819474 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:46.819431 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:43:46 +0000 UTC" deadline="2028-06-03 00:42:14.028216744 +0000 UTC" Oct 30 21:48:46.819474 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:46.819471 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13946h53m27.208750515s" Oct 30 21:48:47.333303 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:47.331633 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs\") pod \"network-metrics-daemon-69j9v\" (UID: \"bd3b421a-9313-4f15-b81f-900717685fd6\") " pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:48:47.333303 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:47.331835 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:48:47.333303 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:47.331916 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs podName:bd3b421a-9313-4f15-b81f-900717685fd6 nodeName:}" failed. No retries permitted until 2026-10-30 21:48:49.331901072 +0000 UTC m=+6.151634608 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs") pod "network-metrics-daemon-69j9v" (UID: "bd3b421a-9313-4f15-b81f-900717685fd6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:48:47.432762 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:47.432721 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret\") pod \"global-pull-secret-syncer-lfq7z\" (UID: \"1d430f7e-00f1-4b9e-9427-3e9971cfff31\") " pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:48:47.432959 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:47.432897 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:48:47.433023 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:47.432967 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret podName:1d430f7e-00f1-4b9e-9427-3e9971cfff31 nodeName:}" failed. No retries permitted until 2026-10-30 21:48:49.432948645 +0000 UTC m=+6.252682186 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret") pod "global-pull-secret-syncer-lfq7z" (UID: "1d430f7e-00f1-4b9e-9427-3e9971cfff31") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:48:47.534438 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:47.534039 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk7l4\" (UniqueName: \"kubernetes.io/projected/550e690a-605e-47a4-ba95-74bcd26e0cc3-kube-api-access-kk7l4\") pod \"network-check-target-6gzkm\" (UID: \"550e690a-605e-47a4-ba95-74bcd26e0cc3\") " pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:48:47.534438 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:47.534270 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:48:47.534438 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:47.534315 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:48:47.534438 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:47.534330 2578 projected.go:196] Error preparing data for projected volume kube-api-access-kk7l4 for pod openshift-network-diagnostics/network-check-target-6gzkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:48:47.534438 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:47.534393 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/550e690a-605e-47a4-ba95-74bcd26e0cc3-kube-api-access-kk7l4 podName:550e690a-605e-47a4-ba95-74bcd26e0cc3 nodeName:}" failed. No retries permitted until 2026-10-30 21:48:49.534372634 +0000 UTC m=+6.354106186 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kk7l4" (UniqueName: "kubernetes.io/projected/550e690a-605e-47a4-ba95-74bcd26e0cc3-kube-api-access-kk7l4") pod "network-check-target-6gzkm" (UID: "550e690a-605e-47a4-ba95-74bcd26e0cc3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:48:47.681340 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:47.681180 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:48:47.681514 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:47.681390 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lfq7z" podUID="1d430f7e-00f1-4b9e-9427-3e9971cfff31" Oct 30 21:48:47.681514 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:47.681505 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:48:47.681637 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:47.681607 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69j9v" podUID="bd3b421a-9313-4f15-b81f-900717685fd6" Oct 30 21:48:47.681696 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:47.681678 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:48:47.681768 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:47.681746 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gzkm" podUID="550e690a-605e-47a4-ba95-74bcd26e0cc3" Oct 30 21:48:47.819864 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:47.819786 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:43:46 +0000 UTC" deadline="2028-07-07 09:49:29.357330186 +0000 UTC" Oct 30 21:48:47.819864 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:47.819854 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14772h0m41.537480247s" Oct 30 21:48:49.356211 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:49.356120 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs\") pod \"network-metrics-daemon-69j9v\" (UID: \"bd3b421a-9313-4f15-b81f-900717685fd6\") " pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:48:49.356734 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:49.356389 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:48:49.357260 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:49.357239 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs podName:bd3b421a-9313-4f15-b81f-900717685fd6 nodeName:}" failed. No retries permitted until 2026-10-30 21:48:53.356436988 +0000 UTC m=+10.176170539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs") pod "network-metrics-daemon-69j9v" (UID: "bd3b421a-9313-4f15-b81f-900717685fd6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:48:49.459578 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:49.458774 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret\") pod \"global-pull-secret-syncer-lfq7z\" (UID: \"1d430f7e-00f1-4b9e-9427-3e9971cfff31\") " pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:48:49.459578 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:49.459009 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:48:49.459578 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:49.459101 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret podName:1d430f7e-00f1-4b9e-9427-3e9971cfff31 nodeName:}" failed. No retries permitted until 2026-10-30 21:48:53.45905571 +0000 UTC m=+10.278789253 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret") pod "global-pull-secret-syncer-lfq7z" (UID: "1d430f7e-00f1-4b9e-9427-3e9971cfff31") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:48:49.560130 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:49.559442 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk7l4\" (UniqueName: \"kubernetes.io/projected/550e690a-605e-47a4-ba95-74bcd26e0cc3-kube-api-access-kk7l4\") pod \"network-check-target-6gzkm\" (UID: \"550e690a-605e-47a4-ba95-74bcd26e0cc3\") " pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:48:49.560130 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:49.559664 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:48:49.560130 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:49.559686 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:48:49.560130 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:49.559700 2578 projected.go:196] Error preparing data for projected volume kube-api-access-kk7l4 for pod openshift-network-diagnostics/network-check-target-6gzkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:48:49.560130 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:49.559764 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/550e690a-605e-47a4-ba95-74bcd26e0cc3-kube-api-access-kk7l4 podName:550e690a-605e-47a4-ba95-74bcd26e0cc3 nodeName:}" failed. No retries permitted until 2026-10-30 21:48:53.559745213 +0000 UTC m=+10.379478754 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kk7l4" (UniqueName: "kubernetes.io/projected/550e690a-605e-47a4-ba95-74bcd26e0cc3-kube-api-access-kk7l4") pod "network-check-target-6gzkm" (UID: "550e690a-605e-47a4-ba95-74bcd26e0cc3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:48:49.682426 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:49.681905 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:48:49.682426 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:49.682066 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69j9v" podUID="bd3b421a-9313-4f15-b81f-900717685fd6" Oct 30 21:48:49.682426 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:49.682158 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:48:49.682426 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:49.682239 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gzkm" podUID="550e690a-605e-47a4-ba95-74bcd26e0cc3" Oct 30 21:48:49.682426 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:49.682289 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:48:49.682426 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:49.682375 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lfq7z" podUID="1d430f7e-00f1-4b9e-9427-3e9971cfff31" Oct 30 21:48:50.734926 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:50.734666 2578 generic.go:356] "Generic (PLEG): container finished" podID="556c8f6eb09b355e5322f215128e2cca" containerID="78c0e6ae83bcc50b9009d1b177396980329d8d01b22ae1f7ab4170f20b29d6eb" exitCode=0 Oct 30 21:48:50.735445 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:50.734879 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal" event={"ID":"556c8f6eb09b355e5322f215128e2cca","Type":"ContainerDied","Data":"78c0e6ae83bcc50b9009d1b177396980329d8d01b22ae1f7ab4170f20b29d6eb"} Oct 30 21:48:50.742040 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:50.741944 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal" event={"ID":"774172d5b7cfcf0d4b41c09fc8c830a1","Type":"ContainerStarted","Data":"b32a86641040e3fc823c6e0e52cf588c29bbbe4d667949d083a0f7addd3979eb"} Oct 30 21:48:50.754089 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:50.753289 2578 generic.go:356] "Generic (PLEG): container finished" podID="ffa64e38-3f1b-49af-913f-139addefa359" containerID="ef7b822b982cd0f1a5bc539eadafc7b0cb6e0d6e3b4057e6de4588839bce0d7f" exitCode=0 Oct 30 21:48:50.754089 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:50.753370 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2t6m" event={"ID":"ffa64e38-3f1b-49af-913f-139addefa359","Type":"ContainerDied","Data":"ef7b822b982cd0f1a5bc539eadafc7b0cb6e0d6e3b4057e6de4588839bce0d7f"} Oct 30 21:48:51.681121 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:51.678810 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:48:51.681121 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:51.678959 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69j9v" podUID="bd3b421a-9313-4f15-b81f-900717685fd6" Oct 30 21:48:51.681121 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:51.679401 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:48:51.681121 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:51.679509 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gzkm" podUID="550e690a-605e-47a4-ba95-74bcd26e0cc3" Oct 30 21:48:51.681121 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:51.679564 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:48:51.681121 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:51.679638 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lfq7z" podUID="1d430f7e-00f1-4b9e-9427-3e9971cfff31" Oct 30 21:48:51.761335 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:51.760801 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-929rz" event={"ID":"64086350-55f6-4da5-a871-3f23c1755371","Type":"ContainerStarted","Data":"232104b4580740852986189a3eab7852c9976fbc781bdcae006dab6ca4627896"} Oct 30 21:48:51.765420 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:51.765287 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal" event={"ID":"556c8f6eb09b355e5322f215128e2cca","Type":"ContainerStarted","Data":"73dbff0e55bccca4e361c26c33d1a7d0eaaeea76cbb8c0193f2243323dd31372"} Oct 30 21:48:51.891203 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:51.890506 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal" podStartSLOduration=5.890489475 podStartE2EDuration="5.890489475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:51.138938888 +0000 UTC m=+7.958672449" watchObservedRunningTime="2026-10-30 21:48:51.890489475 +0000 UTC m=+8.710223090" Oct 30 21:48:52.020963 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:52.020129 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal" podStartSLOduration=7.02011427 podStartE2EDuration="7.02011427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:52.019832825 +0000 UTC m=+8.839566384" watchObservedRunningTime="2026-10-30 21:48:52.02011427 +0000 UTC m=+8.839847830" Oct 30 21:48:52.020963 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:52.020697 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-929rz" podStartSLOduration=2.990235262 podStartE2EDuration="8.020687634s" totalImagesPullingTime="5.030452372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:51.891043657 +0000 UTC m=+8.710777216" watchObservedRunningTime="2026-10-30 21:48:52.020687634 +0000 UTC m=+8.840421197" Oct 30 21:48:52.771606 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:52.771502 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x4655" event={"ID":"ceedf44e-88c1-49cf-901e-75558703c361","Type":"ContainerStarted","Data":"1b833ddf82da1af4da53d0cc7646cfc5ae03a2a477fe88af0aae5840d65321f8"} Oct 30 21:48:52.902063 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:52.902001 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-x4655" podStartSLOduration=2.4914872519999998 podStartE2EDuration="8.901982045s" totalImagesPullingTime="6.410494793s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:52.899510114 +0000 UTC m=+9.719243674" watchObservedRunningTime="2026-10-30 21:48:52.901982045 +0000 UTC m=+9.721715607" Oct 30 21:48:53.245073 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:53.245000 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-929rz" Oct 30 21:48:53.245767 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:53.245739 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-929rz" Oct 30 21:48:53.393103 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:53.392713 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs\") pod \"network-metrics-daemon-69j9v\" (UID: \"bd3b421a-9313-4f15-b81f-900717685fd6\") " pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:48:53.393103 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:53.392924 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:48:53.393103 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:53.392991 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs podName:bd3b421a-9313-4f15-b81f-900717685fd6 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:01.392970975 +0000 UTC m=+18.212704518 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs") pod "network-metrics-daemon-69j9v" (UID: "bd3b421a-9313-4f15-b81f-900717685fd6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:48:53.493747 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:53.493446 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret\") pod \"global-pull-secret-syncer-lfq7z\" (UID: \"1d430f7e-00f1-4b9e-9427-3e9971cfff31\") " pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:48:53.493747 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:53.493636 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:48:53.493747 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:53.493702 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret podName:1d430f7e-00f1-4b9e-9427-3e9971cfff31 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:01.49368329 +0000 UTC m=+18.313416844 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret") pod "global-pull-secret-syncer-lfq7z" (UID: "1d430f7e-00f1-4b9e-9427-3e9971cfff31") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:48:53.594550 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:53.594462 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk7l4\" (UniqueName: \"kubernetes.io/projected/550e690a-605e-47a4-ba95-74bcd26e0cc3-kube-api-access-kk7l4\") pod \"network-check-target-6gzkm\" (UID: \"550e690a-605e-47a4-ba95-74bcd26e0cc3\") " pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:48:53.594688 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:53.594671 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:48:53.594722 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:53.594690 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:48:53.594722 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:53.594703 2578 projected.go:196] Error preparing data for projected volume kube-api-access-kk7l4 for pod openshift-network-diagnostics/network-check-target-6gzkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:48:53.594784 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:53.594760 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/550e690a-605e-47a4-ba95-74bcd26e0cc3-kube-api-access-kk7l4 podName:550e690a-605e-47a4-ba95-74bcd26e0cc3 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:01.594742087 +0000 UTC m=+18.414475635 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kk7l4" (UniqueName: "kubernetes.io/projected/550e690a-605e-47a4-ba95-74bcd26e0cc3-kube-api-access-kk7l4") pod "network-check-target-6gzkm" (UID: "550e690a-605e-47a4-ba95-74bcd26e0cc3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:48:53.685630 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:53.685604 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:48:53.685790 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:53.685764 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69j9v" podUID="bd3b421a-9313-4f15-b81f-900717685fd6" Oct 30 21:48:53.685790 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:53.685674 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:48:53.685916 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:53.685839 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lfq7z" podUID="1d430f7e-00f1-4b9e-9427-3e9971cfff31" Oct 30 21:48:53.685916 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:53.685658 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:48:53.685916 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:53.685895 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gzkm" podUID="550e690a-605e-47a4-ba95-74bcd26e0cc3" Oct 30 21:48:53.774982 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:53.774942 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" event={"ID":"b2f2c486-d661-4d59-981c-0b95a1f66fb4","Type":"ContainerStarted","Data":"095257cbac0780eff5d1de75b7a3d18314c41b9f4194f222d237e632a2da0d39"} Oct 30 21:48:54.776904 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:54.776444 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:48:55.404374 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:55.404331 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:48:55.678800 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:55.678690 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:48:55.678800 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:55.678722 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:48:55.678997 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:55.678690 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:48:55.678997 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:55.678820 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lfq7z" podUID="1d430f7e-00f1-4b9e-9427-3e9971cfff31" Oct 30 21:48:55.678997 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:55.678951 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69j9v" podUID="bd3b421a-9313-4f15-b81f-900717685fd6" Oct 30 21:48:55.679120 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:55.679032 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gzkm" podUID="550e690a-605e-47a4-ba95-74bcd26e0cc3" Oct 30 21:48:55.685929 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:55.685842 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:48:55.404353334Z","UUID":"966c1b70-f68e-4069-82ff-66d604a4fea9","Handler":null,"Name":"","Endpoint":""} Oct 30 21:48:55.687400 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:55.687355 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:48:55.687400 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:55.687381 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:48:55.780978 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:55.780809 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-74zqz" event={"ID":"1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4","Type":"ContainerStarted","Data":"79f66f6c1913611b108cc562c0a5891380fea74cdbec6bb0a927bbddb9b12f47"} Oct 30 21:48:55.783397 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:55.783375 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" event={"ID":"b2f2c486-d661-4d59-981c-0b95a1f66fb4","Type":"ContainerStarted","Data":"5c95ea1919dcd1d5770ac775f0bab8bc8ffeadedf7d95ffcc8c1f9cea10bd0b3"} Oct 30 21:48:55.786291 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:55.786048 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mmdsp" event={"ID":"13fd2c81-54ab-4afa-9afe-68094563bf08","Type":"ContainerStarted","Data":"4f27d1fe5f2a61b10bb5e773bc8f5e40306da22722145fec1400b0c12f2d0d53"} Oct 30 21:48:55.928601 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:55.928545 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-74zqz" podStartSLOduration=1.735844625 podStartE2EDuration="10.928532491s" totalImagesPullingTime="9.192687866s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:48:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:55.928101593 +0000 UTC m=+12.747835149" watchObservedRunningTime="2026-10-30 21:48:55.928532491 +0000 UTC m=+12.748266046" Oct 30 21:48:56.049077 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:56.049010 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-mmdsp" podStartSLOduration=3.050979107 podStartE2EDuration="12.048993158s" totalImagesPullingTime="8.998014051s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:56.048908853 +0000 UTC m=+12.868642412" watchObservedRunningTime="2026-10-30 21:48:56.048993158 +0000 UTC m=+12.868726715" Oct 30 21:48:56.792548 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:56.792509 2578 generic.go:356] "Generic (PLEG): container finished" podID="ffa64e38-3f1b-49af-913f-139addefa359" containerID="8a9cc2cff1cd2dbeedaee92b373dc530a85e2120851aaa05532ed00da25336f1" exitCode=0 Oct 30 21:48:56.793519 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:56.793495 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2t6m" event={"ID":"ffa64e38-3f1b-49af-913f-139addefa359","Type":"ContainerDied","Data":"8a9cc2cff1cd2dbeedaee92b373dc530a85e2120851aaa05532ed00da25336f1"} Oct 30 21:48:57.679073 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:57.679036 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:48:57.679235 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:57.679036 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:48:57.679235 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:57.679167 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:48:57.679235 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:57.679165 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lfq7z" podUID="1d430f7e-00f1-4b9e-9427-3e9971cfff31" Oct 30 21:48:57.679407 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:57.679241 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gzkm" podUID="550e690a-605e-47a4-ba95-74bcd26e0cc3" Oct 30 21:48:57.679407 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:57.679373 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69j9v" podUID="bd3b421a-9313-4f15-b81f-900717685fd6" Oct 30 21:48:57.796413 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:57.796326 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gft82" event={"ID":"9161cdbd-c44f-4a8c-b918-107fc7001f3a","Type":"ContainerStarted","Data":"92987cde3684af563e81aff395194d123f75ce3183586cfc6067ac236ad6f05c"} Oct 30 21:48:57.798566 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:57.798537 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" event={"ID":"b2f2c486-d661-4d59-981c-0b95a1f66fb4","Type":"ContainerStarted","Data":"9389e359fb871ba7068db346afe6242ff796542a89cc5b4333bbc7dc72dfe37e"} Oct 30 21:48:58.038825 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:58.038772 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" podStartSLOduration=3.12994727 podStartE2EDuration="14.038755874s" totalImagesPullingTime="10.908808604s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:48:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:58.038642493 +0000 UTC m=+14.858376050" watchObservedRunningTime="2026-10-30 21:48:58.038755874 +0000 UTC m=+14.858489430" Oct 30 21:48:58.039081 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:58.039043 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gft82" podStartSLOduration=4.662217114 podStartE2EDuration="14.039030076s" totalImagesPullingTime="9.376812962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:57.915432549 +0000 UTC m=+14.735166107" watchObservedRunningTime="2026-10-30 21:48:58.039030076 +0000 UTC m=+14.858763632" Oct 30 21:48:58.803548 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:58.803511 2578 generic.go:356] "Generic (PLEG): container finished" podID="ffa64e38-3f1b-49af-913f-139addefa359" containerID="101d52438331cc44886088fab89f5f41da997f928a88e1c3dd3da632a9b73f49" exitCode=0 Oct 30 21:48:58.804232 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:58.803594 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2t6m" event={"ID":"ffa64e38-3f1b-49af-913f-139addefa359","Type":"ContainerDied","Data":"101d52438331cc44886088fab89f5f41da997f928a88e1c3dd3da632a9b73f49"} Oct 30 21:48:58.804994 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:58.804971 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" event={"ID":"a32b60f2-3000-4fe8-a928-955de8c76c82","Type":"ContainerStarted","Data":"e215b58d037851b16c07e4eac3d0f25f92118bb2453000e2fdcd3b48c79b6b8b"} Oct 30 21:48:59.110918 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:59.110809 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" podStartSLOduration=3.287662562 podStartE2EDuration="15.110793108s" totalImagesPullingTime="11.823130546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:59.110534502 +0000 UTC m=+15.930268060" watchObservedRunningTime="2026-10-30 21:48:59.110793108 +0000 UTC m=+15.930526665" Oct 30 21:48:59.679034 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:59.678844 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:48:59.679224 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:59.678873 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:48:59.679224 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:59.679142 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gzkm" podUID="550e690a-605e-47a4-ba95-74bcd26e0cc3" Oct 30 21:48:59.679224 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:48:59.678924 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:48:59.679224 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:59.679216 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lfq7z" podUID="1d430f7e-00f1-4b9e-9427-3e9971cfff31" Oct 30 21:48:59.679428 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:48:59.679337 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69j9v" podUID="bd3b421a-9313-4f15-b81f-900717685fd6" Oct 30 21:49:00.812032 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:00.811994 2578 generic.go:356] "Generic (PLEG): container finished" podID="ffa64e38-3f1b-49af-913f-139addefa359" containerID="7a0096a9566e03670dfe5fa22ee26965fed73e5678cfca8d098fb29a8013d6d0" exitCode=0 Oct 30 21:49:00.812486 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:00.812047 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2t6m" event={"ID":"ffa64e38-3f1b-49af-913f-139addefa359","Type":"ContainerDied","Data":"7a0096a9566e03670dfe5fa22ee26965fed73e5678cfca8d098fb29a8013d6d0"} Oct 30 21:49:01.458586 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:01.458535 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs\") pod \"network-metrics-daemon-69j9v\" (UID: \"bd3b421a-9313-4f15-b81f-900717685fd6\") " pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:49:01.458817 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:01.458661 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:49:01.458817 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:01.458715 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs podName:bd3b421a-9313-4f15-b81f-900717685fd6 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:17.458701667 +0000 UTC m=+34.278435201 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs") pod "network-metrics-daemon-69j9v" (UID: "bd3b421a-9313-4f15-b81f-900717685fd6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:49:01.559465 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:01.559425 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret\") pod \"global-pull-secret-syncer-lfq7z\" (UID: \"1d430f7e-00f1-4b9e-9427-3e9971cfff31\") " pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:49:01.559659 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:01.559549 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:49:01.559659 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:01.559609 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret podName:1d430f7e-00f1-4b9e-9427-3e9971cfff31 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:17.559597013 +0000 UTC m=+34.379330552 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret") pod "global-pull-secret-syncer-lfq7z" (UID: "1d430f7e-00f1-4b9e-9427-3e9971cfff31") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:49:01.660296 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:01.660238 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk7l4\" (UniqueName: \"kubernetes.io/projected/550e690a-605e-47a4-ba95-74bcd26e0cc3-kube-api-access-kk7l4\") pod \"network-check-target-6gzkm\" (UID: \"550e690a-605e-47a4-ba95-74bcd26e0cc3\") " pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:49:01.660474 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:01.660370 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:49:01.660474 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:01.660392 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:49:01.660474 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:01.660402 2578 projected.go:196] Error preparing data for projected volume kube-api-access-kk7l4 for pod openshift-network-diagnostics/network-check-target-6gzkm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:49:01.660474 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:01.660471 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/550e690a-605e-47a4-ba95-74bcd26e0cc3-kube-api-access-kk7l4 podName:550e690a-605e-47a4-ba95-74bcd26e0cc3 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:17.660452806 +0000 UTC m=+34.480186356 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kk7l4" (UniqueName: "kubernetes.io/projected/550e690a-605e-47a4-ba95-74bcd26e0cc3-kube-api-access-kk7l4") pod "network-check-target-6gzkm" (UID: "550e690a-605e-47a4-ba95-74bcd26e0cc3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:49:01.681094 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:01.681024 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:49:01.681295 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:01.681162 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69j9v" podUID="bd3b421a-9313-4f15-b81f-900717685fd6" Oct 30 21:49:01.681295 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:01.681239 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:49:01.681408 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:01.681333 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gzkm" podUID="550e690a-605e-47a4-ba95-74bcd26e0cc3" Oct 30 21:49:01.681447 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:01.681399 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:49:01.681511 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:01.681489 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lfq7z" podUID="1d430f7e-00f1-4b9e-9427-3e9971cfff31" Oct 30 21:49:02.301220 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:02.301183 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-929rz" Oct 30 21:49:02.301674 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:02.301388 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:49:02.302279 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:02.302192 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-929rz" Oct 30 21:49:02.819068 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:02.819033 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" event={"ID":"7aa1b80f-115e-4708-bc98-88d13ad66860","Type":"ContainerStarted","Data":"75860cf7de2848304c4e2ff1f4859387e571684c7095189a690afa0cc198282d"} Oct 30 21:49:02.819179 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:02.819111 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" event={"ID":"7aa1b80f-115e-4708-bc98-88d13ad66860","Type":"ContainerStarted","Data":"f49609805fc69f58ea3c78aa6c0772053f8fab65c7ffc825e44c5aac682366b6"} Oct 30 21:49:02.819179 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:02.819158 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" event={"ID":"7aa1b80f-115e-4708-bc98-88d13ad66860","Type":"ContainerStarted","Data":"6a56288a6c1b0ba5bbe93df97d4b10f8951cc463141bed99196c085cc9c6b3bd"} Oct 30 21:49:02.819303 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:02.819207 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" event={"ID":"7aa1b80f-115e-4708-bc98-88d13ad66860","Type":"ContainerStarted","Data":"2cd8698505e5ea859e7e256180687339120314dbf120a63f3cc2878ce72951ae"} Oct 30 21:49:03.679421 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:03.679389 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:49:03.680065 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:03.679472 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gzkm" podUID="550e690a-605e-47a4-ba95-74bcd26e0cc3" Oct 30 21:49:03.680065 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:03.679510 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:49:03.680065 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:03.679559 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69j9v" podUID="bd3b421a-9313-4f15-b81f-900717685fd6" Oct 30 21:49:03.680065 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:03.679608 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:49:03.680065 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:03.679941 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lfq7z" podUID="1d430f7e-00f1-4b9e-9427-3e9971cfff31" Oct 30 21:49:03.824355 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:03.824318 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" event={"ID":"7aa1b80f-115e-4708-bc98-88d13ad66860","Type":"ContainerStarted","Data":"1ede1746f36bbcf4dc7049f82b0c86d8aa37cc84a30b2933ed69034a1025aa6f"} Oct 30 21:49:03.824512 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:03.824402 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" event={"ID":"7aa1b80f-115e-4708-bc98-88d13ad66860","Type":"ContainerStarted","Data":"5b88a33315357783f624adc6ac8b2cf216659dbb1853f2ce07008b552fb84c09"} Oct 30 21:49:03.826645 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:03.826614 2578 generic.go:356] "Generic (PLEG): container finished" podID="ffa64e38-3f1b-49af-913f-139addefa359" containerID="fb417171a083de4a68b44d1fca299d43787122ac6e2692bceb3fc060dd630155" exitCode=0 Oct 30 21:49:03.826765 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:03.826657 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2t6m" event={"ID":"ffa64e38-3f1b-49af-913f-139addefa359","Type":"ContainerDied","Data":"fb417171a083de4a68b44d1fca299d43787122ac6e2692bceb3fc060dd630155"} Oct 30 21:49:04.832047 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:04.832003 2578 generic.go:356] "Generic (PLEG): container finished" podID="ffa64e38-3f1b-49af-913f-139addefa359" containerID="22144b84ceaa81b5b23c27729b939d9027ca8acc4583ef62e56fd254c585a374" exitCode=0 Oct 30 21:49:04.832554 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:04.832070 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2t6m" event={"ID":"ffa64e38-3f1b-49af-913f-139addefa359","Type":"ContainerDied","Data":"22144b84ceaa81b5b23c27729b939d9027ca8acc4583ef62e56fd254c585a374"} Oct 30 21:49:05.679695 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:05.679657 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:49:05.679869 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:05.679669 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:49:05.679869 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:05.679784 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:49:05.679869 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:05.679784 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lfq7z" podUID="1d430f7e-00f1-4b9e-9427-3e9971cfff31" Oct 30 21:49:05.679869 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:05.679862 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gzkm" podUID="550e690a-605e-47a4-ba95-74bcd26e0cc3" Oct 30 21:49:05.680088 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:05.679935 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69j9v" podUID="bd3b421a-9313-4f15-b81f-900717685fd6" Oct 30 21:49:05.837666 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:05.837631 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" event={"ID":"7aa1b80f-115e-4708-bc98-88d13ad66860","Type":"ContainerStarted","Data":"7bc211f94bb1c912de0b3e2401dc347fcb0088bbb90b74af171e3f522bda29eb"} Oct 30 21:49:05.840824 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:05.840802 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2t6m" event={"ID":"ffa64e38-3f1b-49af-913f-139addefa359","Type":"ContainerStarted","Data":"5bb6801ec2a3bbe1c6287e0113a3d9437e6726c7245b5ef5ccc8754eb79c9ca4"} Oct 30 21:49:05.977464 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:05.977368 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-w2t6m" podStartSLOduration=8.047121993 podStartE2EDuration="21.977355901s" totalImagesPullingTime="13.930233908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:48:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:05.977276386 +0000 UTC m=+22.797009942" watchObservedRunningTime="2026-10-30 21:49:05.977355901 +0000 UTC m=+22.797089456" Oct 30 21:49:07.678813 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:07.678770 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:49:07.678813 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:07.678796 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:49:07.679383 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:07.678864 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:49:07.679383 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:07.678948 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lfq7z" podUID="1d430f7e-00f1-4b9e-9427-3e9971cfff31" Oct 30 21:49:07.679383 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:07.679104 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gzkm" podUID="550e690a-605e-47a4-ba95-74bcd26e0cc3" Oct 30 21:49:07.679383 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:07.679208 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69j9v" podUID="bd3b421a-9313-4f15-b81f-900717685fd6" Oct 30 21:49:07.848568 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:07.848533 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" event={"ID":"7aa1b80f-115e-4708-bc98-88d13ad66860","Type":"ContainerStarted","Data":"31e90200ca7ecf5cea26eea60afe7ca134c0982b21a10dc4eada5acebd54f3c0"} Oct 30 21:49:07.848818 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:07.848798 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:07.864619 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:07.864592 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:08.133680 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:08.133613 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" podStartSLOduration=7.685575123 podStartE2EDuration="24.133599652s" totalImagesPullingTime="16.448024529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:08.082765712 +0000 UTC m=+24.902499268" watchObservedRunningTime="2026-10-30 21:49:08.133599652 +0000 UTC m=+24.953333198" Oct 30 21:49:08.850654 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:08.850620 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:08.851006 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:08.850722 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:08.865878 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:08.865857 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:09.679488 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:09.679458 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:49:09.679668 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:09.679515 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:49:09.679668 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:09.679520 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:49:09.679668 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:09.679607 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gzkm" podUID="550e690a-605e-47a4-ba95-74bcd26e0cc3" Oct 30 21:49:09.679856 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:09.679760 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69j9v" podUID="bd3b421a-9313-4f15-b81f-900717685fd6" Oct 30 21:49:09.679887 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:09.679858 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lfq7z" podUID="1d430f7e-00f1-4b9e-9427-3e9971cfff31" Oct 30 21:49:09.726769 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:09.726729 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lfq7z"] Oct 30 21:49:09.730383 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:09.730357 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-69j9v"] Oct 30 21:49:09.733863 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:09.733841 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6gzkm"] Oct 30 21:49:09.851812 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:09.851788 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:49:09.851812 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:09.851794 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:49:09.852216 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:09.851788 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:49:09.852216 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:09.851885 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lfq7z" podUID="1d430f7e-00f1-4b9e-9427-3e9971cfff31" Oct 30 21:49:09.852216 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:09.851965 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gzkm" podUID="550e690a-605e-47a4-ba95-74bcd26e0cc3" Oct 30 21:49:09.852335 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:09.852304 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69j9v" podUID="bd3b421a-9313-4f15-b81f-900717685fd6" Oct 30 21:49:11.678951 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:11.678925 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:49:11.678951 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:11.678953 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:49:11.679483 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:11.679036 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69j9v" podUID="bd3b421a-9313-4f15-b81f-900717685fd6" Oct 30 21:49:11.679483 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:11.679094 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gzkm" podUID="550e690a-605e-47a4-ba95-74bcd26e0cc3" Oct 30 21:49:11.679483 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:11.679119 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:49:11.679483 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:11.679223 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lfq7z" podUID="1d430f7e-00f1-4b9e-9427-3e9971cfff31" Oct 30 21:49:13.680163 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:13.680131 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:49:13.680635 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:13.680306 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:49:13.680635 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:13.680323 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69j9v" podUID="bd3b421a-9313-4f15-b81f-900717685fd6" Oct 30 21:49:13.680635 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:13.680420 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:49:13.680635 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:13.680422 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gzkm" podUID="550e690a-605e-47a4-ba95-74bcd26e0cc3" Oct 30 21:49:13.680635 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:13.680503 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lfq7z" podUID="1d430f7e-00f1-4b9e-9427-3e9971cfff31" Oct 30 21:49:15.679813 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:15.679640 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:49:15.680224 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:15.679675 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:49:15.680224 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:15.679888 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lfq7z" podUID="1d430f7e-00f1-4b9e-9427-3e9971cfff31" Oct 30 21:49:15.680224 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:15.679994 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gzkm" podUID="550e690a-605e-47a4-ba95-74bcd26e0cc3" Oct 30 21:49:15.680224 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:15.679715 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:49:15.680224 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:15.680121 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69j9v" podUID="bd3b421a-9313-4f15-b81f-900717685fd6" Oct 30 21:49:16.032936 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.032866 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeReady" Oct 30 21:49:16.033098 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.032995 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 21:49:16.151813 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.151776 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-k6p2v"] Oct 30 21:49:16.231704 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.231666 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2xctf"] Oct 30 21:49:16.231865 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.231792 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-k6p2v" Oct 30 21:49:16.252435 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.252400 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6pjx4"] Oct 30 21:49:16.252604 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.252527 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-2xctf" Oct 30 21:49:16.263787 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.263750 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-5xd2z"] Oct 30 21:49:16.263910 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.263876 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6pjx4" Oct 30 21:49:16.277851 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.277830 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq"] Oct 30 21:49:16.277974 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.277919 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-5xd2z" Oct 30 21:49:16.285582 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.285554 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 21:49:16.285689 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.285622 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 21:49:16.292034 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.292014 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-jrhw4"] Oct 30 21:49:16.292140 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.292112 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq" Oct 30 21:49:16.303048 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.302103 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl"] Oct 30 21:49:16.303359 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.303337 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:16.313672 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.313631 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-929rz_64086350-55f6-4da5-a871-3f23c1755371/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:16.314307 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.314287 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68775ff9b8-rkd8f"] Oct 30 21:49:16.314385 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.314352 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl" Oct 30 21:49:16.327440 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.327415 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55fdbc6f6f-ngftj"] Oct 30 21:49:16.327582 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.327451 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:16.335297 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.335275 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 21:49:16.335423 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.335384 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-qzl5d\"" Oct 30 21:49:16.339543 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.339524 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-8w8b5"] Oct 30 21:49:16.339666 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.339650 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:16.347396 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.347339 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 21:49:16.359837 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.359809 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-76684dbbf-8mwh7"] Oct 30 21:49:16.359975 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.359933 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:49:16.362207 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.362186 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2b29\" (UniqueName: \"kubernetes.io/projected/8bf0fd8f-2294-4729-9f49-a151de8627aa-kube-api-access-r2b29\") pod \"kas-connection-checker-685895c7c9-2xctf\" (UID: \"8bf0fd8f-2294-4729-9f49-a151de8627aa\") " pod="kube-system/kas-connection-checker-685895c7c9-2xctf" Oct 30 21:49:16.362324 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.362214 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr7zd\" (UniqueName: \"kubernetes.io/projected/0dce5b38-e1a7-41e3-8100-96af4e3c2b06-kube-api-access-jr7zd\") pod \"kas-connection-checker-685895c7c9-6pjx4\" (UID: \"0dce5b38-e1a7-41e3-8100-96af4e3c2b06\") " pod="kube-system/kas-connection-checker-685895c7c9-6pjx4" Oct 30 21:49:16.362363 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.362323 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thhdq\" (UniqueName: \"kubernetes.io/projected/9df43c88-a01f-4ec7-a385-d5afcf6f9e6c-kube-api-access-thhdq\") pod \"kas-connection-checker-685895c7c9-k6p2v\" (UID: \"9df43c88-a01f-4ec7-a385-d5afcf6f9e6c\") " pod="kube-system/kas-connection-checker-685895c7c9-k6p2v" Oct 30 21:49:16.362407 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.362385 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/47917159-29d7-45b9-bbf2-13cef4408de9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5xd2z\" (UID: \"47917159-29d7-45b9-bbf2-13cef4408de9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xd2z" Oct 30 21:49:16.362437 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.362423 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/47917159-29d7-45b9-bbf2-13cef4408de9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5xd2z\" (UID: \"47917159-29d7-45b9-bbf2-13cef4408de9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xd2z" Oct 30 21:49:16.381536 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.381505 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-55pjz"] Oct 30 21:49:16.381656 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.381558 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:16.386479 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.386440 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 21:49:16.386584 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.386501 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 21:49:16.386584 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.386516 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:49:16.386584 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.386527 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:49:16.386584 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.386520 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:49:16.386744 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.386728 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:49:16.386979 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.386964 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:49:16.387017 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.386988 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:49:16.387435 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.387310 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 21:49:16.402190 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.402166 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 21:49:16.403875 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.403858 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-88kh6"] Oct 30 21:49:16.403998 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.403984 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-55pjz" Oct 30 21:49:16.418442 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.418423 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p54hn"] Oct 30 21:49:16.418549 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.418541 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88kh6" Oct 30 21:49:16.440522 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.440489 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:49:16.442466 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.442438 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:49:16.442665 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.442649 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-989z4"] Oct 30 21:49:16.442792 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.442776 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p54hn" Oct 30 21:49:16.446277 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.446240 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:49:16.446760 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.446742 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 21:49:16.446760 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.446770 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ntfhf\"" Oct 30 21:49:16.446915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.446804 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-4wbzb\"" Oct 30 21:49:16.446915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.446804 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-7fn9v\"" Oct 30 21:49:16.446915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.446823 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 21:49:16.446915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.446829 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 21:49:16.446915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.446840 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 21:49:16.446915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.446848 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 21:49:16.446915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.446838 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:49:16.446915 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.446912 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:49:16.447190 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.446942 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8rvzh\"" Oct 30 21:49:16.447190 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.446943 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-sjphq\"" Oct 30 21:49:16.447190 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.446970 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-b95t7\"" Oct 30 21:49:16.447190 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.447054 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 21:49:16.462815 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.462796 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-zxjct\"" Oct 30 21:49:16.462908 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.462894 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9lcg\" (UniqueName: \"kubernetes.io/projected/fc3ff956-6466-46e3-b734-13a3ac148d36-kube-api-access-k9lcg\") pod \"cluster-samples-operator-6ddb6c8b94-w7pwq\" (UID: \"fc3ff956-6466-46e3-b734-13a3ac148d36\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq" Oct 30 21:49:16.462959 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.462922 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51-config\") pod \"console-operator-7cd56b8dbc-8w8b5\" (UID: \"5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:49:16.462959 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.462942 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/84bfdb37-6346-4384-8d97-d43aba6f03fe-image-registry-private-configuration\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:16.463022 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.462996 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2b29\" (UniqueName: \"kubernetes.io/projected/8bf0fd8f-2294-4729-9f49-a151de8627aa-kube-api-access-r2b29\") pod \"kas-connection-checker-685895c7c9-2xctf\" (UID: \"8bf0fd8f-2294-4729-9f49-a151de8627aa\") " pod="kube-system/kas-connection-checker-685895c7c9-2xctf" Oct 30 21:49:16.463055 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463022 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-tls\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:16.463086 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463050 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:16.463117 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463084 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-certificates\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:16.463148 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463117 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51-serving-cert\") pod \"console-operator-7cd56b8dbc-8w8b5\" (UID: \"5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:49:16.463184 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463152 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de2247d3-e5ef-4a77-b087-8622bec117bc-trusted-ca\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:16.463184 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463174 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c0cece9c-2008-49ce-87de-4804d4e7de08-snapshots\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:16.463298 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463284 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-w7pwq\" (UID: \"fc3ff956-6466-46e3-b734-13a3ac148d36\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq" Oct 30 21:49:16.463351 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463305 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vss2v\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-kube-api-access-vss2v\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:16.463351 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463341 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkjjr\" (UniqueName: \"kubernetes.io/projected/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51-kube-api-access-lkjjr\") pod \"console-operator-7cd56b8dbc-8w8b5\" (UID: \"5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:49:16.463430 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463384 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thhdq\" (UniqueName: \"kubernetes.io/projected/9df43c88-a01f-4ec7-a385-d5afcf6f9e6c-kube-api-access-thhdq\") pod \"kas-connection-checker-685895c7c9-k6p2v\" (UID: \"9df43c88-a01f-4ec7-a385-d5afcf6f9e6c\") " pod="kube-system/kas-connection-checker-685895c7c9-k6p2v" Oct 30 21:49:16.463478 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463468 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 21:49:16.463478 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463473 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-bound-sa-token\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:16.463566 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463490 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/84bfdb37-6346-4384-8d97-d43aba6f03fe-ca-trust-extracted\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:16.463566 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463511 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/84bfdb37-6346-4384-8d97-d43aba6f03fe-installation-pull-secrets\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:16.463566 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463540 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95097323-8907-45bc-8967-0de718adefbd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hmttl\" (UID: \"95097323-8907-45bc-8967-0de718adefbd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl" Oct 30 21:49:16.463699 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463579 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/47917159-29d7-45b9-bbf2-13cef4408de9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5xd2z\" (UID: \"47917159-29d7-45b9-bbf2-13cef4408de9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xd2z" Oct 30 21:49:16.463699 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463599 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/95097323-8907-45bc-8967-0de718adefbd-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-hmttl\" (UID: \"95097323-8907-45bc-8967-0de718adefbd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl" Oct 30 21:49:16.463699 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463619 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58wrk\" (UniqueName: \"kubernetes.io/projected/11ba3d2b-2d12-458c-9984-1dd82c275e10-kube-api-access-58wrk\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:16.463699 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463635 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51-trusted-ca\") pod \"console-operator-7cd56b8dbc-8w8b5\" (UID: \"5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:49:16.463820 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463739 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-certificates\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:16.463820 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463755 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-bound-sa-token\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:16.463820 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463774 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7zd\" (UniqueName: \"kubernetes.io/projected/0dce5b38-e1a7-41e3-8100-96af4e3c2b06-kube-api-access-jr7zd\") pod \"kas-connection-checker-685895c7c9-6pjx4\" (UID: \"0dce5b38-e1a7-41e3-8100-96af4e3c2b06\") " pod="kube-system/kas-connection-checker-685895c7c9-6pjx4" Oct 30 21:49:16.463820 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463789 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p75b\" (UniqueName: \"kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-kube-api-access-8p75b\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:16.463820 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463806 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0cece9c-2008-49ce-87de-4804d4e7de08-service-ca-bundle\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:16.464067 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463838 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0cece9c-2008-49ce-87de-4804d4e7de08-tmp\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:16.464067 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463880 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0cece9c-2008-49ce-87de-4804d4e7de08-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:16.464067 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463914 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j92gp\" (UniqueName: \"kubernetes.io/projected/c0cece9c-2008-49ce-87de-4804d4e7de08-kube-api-access-j92gp\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:16.464067 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.463984 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0cece9c-2008-49ce-87de-4804d4e7de08-serving-cert\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:16.464067 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.464015 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-default-certificate\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:16.464067 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.464052 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-stats-auth\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:16.464383 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.464074 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/de2247d3-e5ef-4a77-b087-8622bec117bc-installation-pull-secrets\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:16.464383 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.464090 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-w7pwq\" (UID: \"fc3ff956-6466-46e3-b734-13a3ac148d36\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq" Oct 30 21:49:16.464383 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.464109 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j58ln\" (UniqueName: \"kubernetes.io/projected/95097323-8907-45bc-8967-0de718adefbd-kube-api-access-j58ln\") pod \"cluster-monitoring-operator-884576c57-hmttl\" (UID: \"95097323-8907-45bc-8967-0de718adefbd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl" Oct 30 21:49:16.464383 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.464155 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ba3d2b-2d12-458c-9984-1dd82c275e10-service-ca-bundle\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:16.464383 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.464159 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/47917159-29d7-45b9-bbf2-13cef4408de9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5xd2z\" (UID: \"47917159-29d7-45b9-bbf2-13cef4408de9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xd2z" Oct 30 21:49:16.464383 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.464173 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-metrics-certs\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:16.464383 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.464203 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/47917159-29d7-45b9-bbf2-13cef4408de9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5xd2z\" (UID: \"47917159-29d7-45b9-bbf2-13cef4408de9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xd2z" Oct 30 21:49:16.464383 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.464244 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/de2247d3-e5ef-4a77-b087-8622bec117bc-image-registry-private-configuration\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:16.464383 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.464286 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/de2247d3-e5ef-4a77-b087-8622bec117bc-ca-trust-extracted\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:16.464383 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:16.464324 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:49:16.464383 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.464368 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-p8hw6"] Oct 30 21:49:16.464383 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.464373 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/84bfdb37-6346-4384-8d97-d43aba6f03fe-trusted-ca\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:16.464383 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:16.464390 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47917159-29d7-45b9-bbf2-13cef4408de9-networking-console-plugin-cert podName:47917159-29d7-45b9-bbf2-13cef4408de9 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:16.964374328 +0000 UTC m=+33.784107879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/47917159-29d7-45b9-bbf2-13cef4408de9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5xd2z" (UID: "47917159-29d7-45b9-bbf2-13cef4408de9") : secret "networking-console-plugin-cert" not found Oct 30 21:49:16.464903 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.464518 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-989z4" Oct 30 21:49:16.478300 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.478280 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-k6p2v"] Oct 30 21:49:16.478676 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.478313 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2xctf"] Oct 30 21:49:16.478676 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.478330 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6pjx4"] Oct 30 21:49:16.478676 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.478331 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:16.478676 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.478345 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b6fbt"] Oct 30 21:49:16.486313 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.486294 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 21:49:16.486541 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.486326 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:49:16.486541 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.486326 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 21:49:16.486541 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.486382 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 21:49:16.486541 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.486501 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:49:16.486541 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.486517 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:49:16.486541 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.486514 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 21:49:16.486541 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.486523 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 21:49:16.486911 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.486697 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 21:49:16.486911 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.486712 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:49:16.486911 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.486725 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:49:16.486911 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.486780 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 21:49:16.489413 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.489397 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 21:49:16.496274 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.496243 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 21:49:16.496377 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.496364 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:49:16.504556 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.504536 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-5xd2z"] Oct 30 21:49:16.504676 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.504573 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq"] Oct 30 21:49:16.504676 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.504603 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-jrhw4"] Oct 30 21:49:16.504676 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.504624 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55fdbc6f6f-ngftj"] Oct 30 21:49:16.504676 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.504646 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-55pjz"] Oct 30 21:49:16.504676 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.504667 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p8hw6"] Oct 30 21:49:16.504912 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.504694 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p54hn"] Oct 30 21:49:16.504912 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.504715 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl"] Oct 30 21:49:16.504912 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.504723 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b6fbt" Oct 30 21:49:16.504912 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.504734 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68775ff9b8-rkd8f"] Oct 30 21:49:16.504912 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.504756 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b6fbt"] Oct 30 21:49:16.504912 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.504779 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-76684dbbf-8mwh7"] Oct 30 21:49:16.504912 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.504800 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-88kh6"] Oct 30 21:49:16.504912 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.504818 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-8w8b5"] Oct 30 21:49:16.504912 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.504839 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-989z4"] Oct 30 21:49:16.527233 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.527207 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thhdq\" (UniqueName: \"kubernetes.io/projected/9df43c88-a01f-4ec7-a385-d5afcf6f9e6c-kube-api-access-thhdq\") pod \"kas-connection-checker-685895c7c9-k6p2v\" (UID: \"9df43c88-a01f-4ec7-a385-d5afcf6f9e6c\") " pod="kube-system/kas-connection-checker-685895c7c9-k6p2v" Oct 30 21:49:16.527385 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.527244 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2b29\" (UniqueName: \"kubernetes.io/projected/8bf0fd8f-2294-4729-9f49-a151de8627aa-kube-api-access-r2b29\") pod \"kas-connection-checker-685895c7c9-2xctf\" (UID: \"8bf0fd8f-2294-4729-9f49-a151de8627aa\") " pod="kube-system/kas-connection-checker-685895c7c9-2xctf" Oct 30 21:49:16.527634 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.527614 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr7zd\" (UniqueName: \"kubernetes.io/projected/0dce5b38-e1a7-41e3-8100-96af4e3c2b06-kube-api-access-jr7zd\") pod \"kas-connection-checker-685895c7c9-6pjx4\" (UID: \"0dce5b38-e1a7-41e3-8100-96af4e3c2b06\") " pod="kube-system/kas-connection-checker-685895c7c9-6pjx4" Oct 30 21:49:16.538806 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.538744 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 21:49:16.539024 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.539005 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:49:16.539156 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.539144 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-52nbt\"" Oct 30 21:49:16.539238 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.539156 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-d8p2x\"" Oct 30 21:49:16.539238 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.539159 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-x2j52\"" Oct 30 21:49:16.539238 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.539163 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-w6rhz\"" Oct 30 21:49:16.555332 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.555303 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-k6p2v" Oct 30 21:49:16.559717 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.559691 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:49:16.559817 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.559801 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8lg5r\"" Oct 30 21:49:16.565115 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.565091 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-w7pwq\" (UID: \"fc3ff956-6466-46e3-b734-13a3ac148d36\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq" Oct 30 21:49:16.565220 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.565119 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgd6c\" (UniqueName: \"kubernetes.io/projected/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-kube-api-access-hgd6c\") pod \"dns-default-p8hw6\" (UID: \"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3\") " pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:16.565220 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.565139 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vss2v\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-kube-api-access-vss2v\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:16.565220 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.565158 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkjjr\" (UniqueName: \"kubernetes.io/projected/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51-kube-api-access-lkjjr\") pod \"console-operator-7cd56b8dbc-8w8b5\" (UID: \"5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:49:16.565220 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.565189 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8258\" (UniqueName: \"kubernetes.io/projected/35a08208-7245-4e1b-9c0b-9e825a808365-kube-api-access-h8258\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-55pjz\" (UID: \"35a08208-7245-4e1b-9c0b-9e825a808365\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-55pjz" Oct 30 21:49:16.565417 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.565226 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-tmp-dir\") pod \"dns-default-p8hw6\" (UID: \"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3\") " pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:16.565417 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.565266 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-bound-sa-token\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:16.565417 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.565314 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/84bfdb37-6346-4384-8d97-d43aba6f03fe-ca-trust-extracted\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:16.565417 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.565378 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/84bfdb37-6346-4384-8d97-d43aba6f03fe-installation-pull-secrets\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:16.565641 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.565542 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95097323-8907-45bc-8967-0de718adefbd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hmttl\" (UID: \"95097323-8907-45bc-8967-0de718adefbd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl" Oct 30 21:49:16.565641 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.565575 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgccp\" (UniqueName: \"kubernetes.io/projected/11109f61-f4f0-4253-aacb-886c353a5772-kube-api-access-vgccp\") pod \"service-ca-operator-544d749c8c-88kh6\" (UID: \"11109f61-f4f0-4253-aacb-886c353a5772\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88kh6" Oct 30 21:49:16.565641 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.565636 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/84bfdb37-6346-4384-8d97-d43aba6f03fe-ca-trust-extracted\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:16.565795 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:16.565675 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:49:16.565795 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.565721 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/95097323-8907-45bc-8967-0de718adefbd-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-hmttl\" (UID: \"95097323-8907-45bc-8967-0de718adefbd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl" Oct 30 21:49:16.565795 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:16.565772 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95097323-8907-45bc-8967-0de718adefbd-cluster-monitoring-operator-tls podName:95097323-8907-45bc-8967-0de718adefbd nodeName:}" failed. No retries permitted until 2026-10-30 21:49:17.065711934 +0000 UTC m=+33.885445470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/95097323-8907-45bc-8967-0de718adefbd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-hmttl" (UID: "95097323-8907-45bc-8967-0de718adefbd") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:49:16.565964 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.565803 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67x9l\" (UniqueName: \"kubernetes.io/projected/226c41d5-de84-4950-978f-a1f0b4cbc319-kube-api-access-67x9l\") pod \"network-check-source-6f78d59f99-989z4\" (UID: \"226c41d5-de84-4950-978f-a1f0b4cbc319\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-989z4" Oct 30 21:49:16.565964 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.565834 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58wrk\" (UniqueName: \"kubernetes.io/projected/11ba3d2b-2d12-458c-9984-1dd82c275e10-kube-api-access-58wrk\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:16.565964 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.565852 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-w7pwq\" (UID: \"fc3ff956-6466-46e3-b734-13a3ac148d36\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq" Oct 30 21:49:16.565964 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.565865 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51-trusted-ca\") pod \"console-operator-7cd56b8dbc-8w8b5\" (UID: \"5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:49:16.565964 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.565904 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11109f61-f4f0-4253-aacb-886c353a5772-serving-cert\") pod \"service-ca-operator-544d749c8c-88kh6\" (UID: \"11109f61-f4f0-4253-aacb-886c353a5772\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88kh6" Oct 30 21:49:16.565964 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.565943 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-certificates\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:16.566440 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.565968 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-bound-sa-token\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:16.566440 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.566022 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p75b\" (UniqueName: \"kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-kube-api-access-8p75b\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:16.566440 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.566052 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0cece9c-2008-49ce-87de-4804d4e7de08-service-ca-bundle\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:16.566440 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.566081 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0cece9c-2008-49ce-87de-4804d4e7de08-tmp\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:16.566440 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.566112 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0cece9c-2008-49ce-87de-4804d4e7de08-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:16.566440 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.566139 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j92gp\" (UniqueName: \"kubernetes.io/projected/c0cece9c-2008-49ce-87de-4804d4e7de08-kube-api-access-j92gp\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:16.566440 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.566182 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0cece9c-2008-49ce-87de-4804d4e7de08-serving-cert\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:16.566440 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.566208 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11109f61-f4f0-4253-aacb-886c353a5772-config\") pod \"service-ca-operator-544d749c8c-88kh6\" (UID: \"11109f61-f4f0-4253-aacb-886c353a5772\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88kh6" Oct 30 21:49:16.566440 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.566236 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-default-certificate\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:16.566440 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.566279 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-metrics-tls\") pod \"dns-default-p8hw6\" (UID: \"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3\") " pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:16.566440 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.566306 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-stats-auth\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:16.566440 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.566334 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/de2247d3-e5ef-4a77-b087-8622bec117bc-installation-pull-secrets\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:16.566440 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.566362 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-w7pwq\" (UID: \"fc3ff956-6466-46e3-b734-13a3ac148d36\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq" Oct 30 21:49:16.566440 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.566396 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j58ln\" (UniqueName: \"kubernetes.io/projected/95097323-8907-45bc-8967-0de718adefbd-kube-api-access-j58ln\") pod \"cluster-monitoring-operator-884576c57-hmttl\" (UID: \"95097323-8907-45bc-8967-0de718adefbd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl" Oct 30 21:49:16.566440 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.566420 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/35a08208-7245-4e1b-9c0b-9e825a808365-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-55pjz\" (UID: \"35a08208-7245-4e1b-9c0b-9e825a808365\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-55pjz" Oct 30 21:49:16.566440 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.566444 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ba3d2b-2d12-458c-9984-1dd82c275e10-service-ca-bundle\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:16.567206 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.566466 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-metrics-certs\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:16.567206 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.566492 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8ctq\" (UniqueName: \"kubernetes.io/projected/4579f026-39bd-465c-b0db-41676370ba55-kube-api-access-l8ctq\") pod \"volume-data-source-validator-58c9f5f46d-p54hn\" (UID: \"4579f026-39bd-465c-b0db-41676370ba55\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p54hn" Oct 30 21:49:16.567206 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.566535 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-certificates\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:16.567206 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.566544 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/de2247d3-e5ef-4a77-b087-8622bec117bc-image-registry-private-configuration\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:16.567206 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.566574 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/de2247d3-e5ef-4a77-b087-8622bec117bc-ca-trust-extracted\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:16.567206 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:16.566639 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:49:16.567206 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:16.566699 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-tls podName:fc3ff956-6466-46e3-b734-13a3ac148d36 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:17.066684387 +0000 UTC m=+33.886417938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-w7pwq" (UID: "fc3ff956-6466-46e3-b734-13a3ac148d36") : secret "samples-operator-tls" not found Oct 30 21:49:16.567206 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.566757 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0cece9c-2008-49ce-87de-4804d4e7de08-service-ca-bundle\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:16.567206 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.566852 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51-trusted-ca\") pod \"console-operator-7cd56b8dbc-8w8b5\" (UID: \"5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:49:16.567206 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.566881 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/de2247d3-e5ef-4a77-b087-8622bec117bc-ca-trust-extracted\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:16.567206 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.566960 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0cece9c-2008-49ce-87de-4804d4e7de08-tmp\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:16.567206 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:16.567199 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/11ba3d2b-2d12-458c-9984-1dd82c275e10-service-ca-bundle podName:11ba3d2b-2d12-458c-9984-1dd82c275e10 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:17.067183804 +0000 UTC m=+33.886917354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/11ba3d2b-2d12-458c-9984-1dd82c275e10-service-ca-bundle") pod "router-default-76684dbbf-8mwh7" (UID: "11ba3d2b-2d12-458c-9984-1dd82c275e10") : configmap references non-existent config key: service-ca.crt Oct 30 21:49:16.567842 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:16.567313 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:49:16.567842 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:16.567357 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-metrics-certs podName:11ba3d2b-2d12-458c-9984-1dd82c275e10 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:17.067344746 +0000 UTC m=+33.887078281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-metrics-certs") pod "router-default-76684dbbf-8mwh7" (UID: "11ba3d2b-2d12-458c-9984-1dd82c275e10") : secret "router-metrics-certs-default" not found Oct 30 21:49:16.567842 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.567572 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0cece9c-2008-49ce-87de-4804d4e7de08-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:16.567842 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.567809 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/95097323-8907-45bc-8967-0de718adefbd-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-hmttl\" (UID: \"95097323-8907-45bc-8967-0de718adefbd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl" Oct 30 21:49:16.568026 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.567857 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/84bfdb37-6346-4384-8d97-d43aba6f03fe-trusted-ca\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:16.568026 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.567894 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9lcg\" (UniqueName: \"kubernetes.io/projected/fc3ff956-6466-46e3-b734-13a3ac148d36-kube-api-access-k9lcg\") pod \"cluster-samples-operator-6ddb6c8b94-w7pwq\" (UID: \"fc3ff956-6466-46e3-b734-13a3ac148d36\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq" Oct 30 21:49:16.568026 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.567918 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51-config\") pod \"console-operator-7cd56b8dbc-8w8b5\" (UID: \"5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:49:16.568026 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.567948 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-config-volume\") pod \"dns-default-p8hw6\" (UID: \"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3\") " pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:16.568026 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.567976 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/84bfdb37-6346-4384-8d97-d43aba6f03fe-image-registry-private-configuration\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:16.568026 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.568005 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-tls\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:16.568325 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.568032 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:16.568325 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.568066 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-certificates\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:16.568325 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.568095 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51-serving-cert\") pod \"console-operator-7cd56b8dbc-8w8b5\" (UID: \"5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:49:16.568325 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.568126 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/35a08208-7245-4e1b-9c0b-9e825a808365-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-55pjz\" (UID: \"35a08208-7245-4e1b-9c0b-9e825a808365\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-55pjz" Oct 30 21:49:16.568325 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.568150 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de2247d3-e5ef-4a77-b087-8622bec117bc-trusted-ca\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:16.568325 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.568190 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c0cece9c-2008-49ce-87de-4804d4e7de08-snapshots\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:16.568620 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.568542 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/84bfdb37-6346-4384-8d97-d43aba6f03fe-installation-pull-secrets\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:16.568671 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:16.568639 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:49:16.568671 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:16.568653 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68775ff9b8-rkd8f: secret "image-registry-tls" not found Oct 30 21:49:16.568769 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:16.568708 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-tls podName:de2247d3-e5ef-4a77-b087-8622bec117bc nodeName:}" failed. No retries permitted until 2026-10-30 21:49:17.068691294 +0000 UTC m=+33.888424830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-tls") pod "image-registry-68775ff9b8-rkd8f" (UID: "de2247d3-e5ef-4a77-b087-8622bec117bc") : secret "image-registry-tls" not found Oct 30 21:49:16.568835 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.568769 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c0cece9c-2008-49ce-87de-4804d4e7de08-snapshots\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:16.568893 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:16.568864 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:49:16.568893 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:16.568877 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55fdbc6f6f-ngftj: secret "image-registry-tls" not found Oct 30 21:49:16.568989 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:16.568918 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls podName:84bfdb37-6346-4384-8d97-d43aba6f03fe nodeName:}" failed. No retries permitted until 2026-10-30 21:49:17.068902505 +0000 UTC m=+33.888636053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls") pod "image-registry-55fdbc6f6f-ngftj" (UID: "84bfdb37-6346-4384-8d97-d43aba6f03fe") : secret "image-registry-tls" not found Oct 30 21:49:16.569450 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.569395 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-certificates\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:16.569644 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.569473 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0cece9c-2008-49ce-87de-4804d4e7de08-serving-cert\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:16.569644 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.569488 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/84bfdb37-6346-4384-8d97-d43aba6f03fe-trusted-ca\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:16.570115 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.570071 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51-config\") pod \"console-operator-7cd56b8dbc-8w8b5\" (UID: \"5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:49:16.570739 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.570713 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/de2247d3-e5ef-4a77-b087-8622bec117bc-installation-pull-secrets\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:16.570830 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.570724 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de2247d3-e5ef-4a77-b087-8622bec117bc-trusted-ca\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:16.570887 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.570833 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/de2247d3-e5ef-4a77-b087-8622bec117bc-image-registry-private-configuration\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:16.571531 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.571506 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-default-certificate\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:16.572094 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.572069 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/84bfdb37-6346-4384-8d97-d43aba6f03fe-image-registry-private-configuration\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:16.572231 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.572198 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-2xctf" Oct 30 21:49:16.572608 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.572590 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51-serving-cert\") pod \"console-operator-7cd56b8dbc-8w8b5\" (UID: \"5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:49:16.573001 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.572983 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-stats-auth\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:16.584798 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.584777 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6pjx4" Oct 30 21:49:16.588181 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.588166 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:49:16.588383 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.588363 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:49:16.588579 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.588564 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:49:16.588579 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.588567 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:49:16.588686 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.588607 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:49:16.632907 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.632871 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-bound-sa-token\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:16.633489 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.633464 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-bound-sa-token\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:16.636156 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.636125 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j58ln\" (UniqueName: \"kubernetes.io/projected/95097323-8907-45bc-8967-0de718adefbd-kube-api-access-j58ln\") pod \"cluster-monitoring-operator-884576c57-hmttl\" (UID: \"95097323-8907-45bc-8967-0de718adefbd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl" Oct 30 21:49:16.636770 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.636746 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j92gp\" (UniqueName: \"kubernetes.io/projected/c0cece9c-2008-49ce-87de-4804d4e7de08-kube-api-access-j92gp\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:16.637419 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.637399 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p75b\" (UniqueName: \"kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-kube-api-access-8p75b\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:16.637706 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.637686 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkjjr\" (UniqueName: \"kubernetes.io/projected/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51-kube-api-access-lkjjr\") pod \"console-operator-7cd56b8dbc-8w8b5\" (UID: \"5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:49:16.638146 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.638132 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58wrk\" (UniqueName: \"kubernetes.io/projected/11ba3d2b-2d12-458c-9984-1dd82c275e10-kube-api-access-58wrk\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:16.638312 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.638290 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vss2v\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-kube-api-access-vss2v\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:16.640513 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.640468 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9lcg\" (UniqueName: \"kubernetes.io/projected/fc3ff956-6466-46e3-b734-13a3ac148d36-kube-api-access-k9lcg\") pod \"cluster-samples-operator-6ddb6c8b94-w7pwq\" (UID: \"fc3ff956-6466-46e3-b734-13a3ac148d36\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq" Oct 30 21:49:16.669370 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.669334 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgccp\" (UniqueName: \"kubernetes.io/projected/11109f61-f4f0-4253-aacb-886c353a5772-kube-api-access-vgccp\") pod \"service-ca-operator-544d749c8c-88kh6\" (UID: \"11109f61-f4f0-4253-aacb-886c353a5772\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88kh6" Oct 30 21:49:16.669501 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.669384 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67x9l\" (UniqueName: \"kubernetes.io/projected/226c41d5-de84-4950-978f-a1f0b4cbc319-kube-api-access-67x9l\") pod \"network-check-source-6f78d59f99-989z4\" (UID: \"226c41d5-de84-4950-978f-a1f0b4cbc319\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-989z4" Oct 30 21:49:16.669501 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.669407 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11109f61-f4f0-4253-aacb-886c353a5772-serving-cert\") pod \"service-ca-operator-544d749c8c-88kh6\" (UID: \"11109f61-f4f0-4253-aacb-886c353a5772\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88kh6" Oct 30 21:49:16.669501 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.669442 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11109f61-f4f0-4253-aacb-886c353a5772-config\") pod \"service-ca-operator-544d749c8c-88kh6\" (UID: \"11109f61-f4f0-4253-aacb-886c353a5772\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88kh6" Oct 30 21:49:16.669501 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.669461 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn7gh\" (UniqueName: \"kubernetes.io/projected/c7fe5efb-7029-41b8-8bbe-f37f49410f27-kube-api-access-fn7gh\") pod \"ingress-canary-b6fbt\" (UID: \"c7fe5efb-7029-41b8-8bbe-f37f49410f27\") " pod="openshift-ingress-canary/ingress-canary-b6fbt" Oct 30 21:49:16.669501 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.669476 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-metrics-tls\") pod \"dns-default-p8hw6\" (UID: \"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3\") " pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:16.669501 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.669491 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7fe5efb-7029-41b8-8bbe-f37f49410f27-cert\") pod \"ingress-canary-b6fbt\" (UID: \"c7fe5efb-7029-41b8-8bbe-f37f49410f27\") " pod="openshift-ingress-canary/ingress-canary-b6fbt" Oct 30 21:49:16.669713 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.669522 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/35a08208-7245-4e1b-9c0b-9e825a808365-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-55pjz\" (UID: \"35a08208-7245-4e1b-9c0b-9e825a808365\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-55pjz" Oct 30 21:49:16.669713 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.669552 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8ctq\" (UniqueName: \"kubernetes.io/projected/4579f026-39bd-465c-b0db-41676370ba55-kube-api-access-l8ctq\") pod \"volume-data-source-validator-58c9f5f46d-p54hn\" (UID: \"4579f026-39bd-465c-b0db-41676370ba55\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p54hn" Oct 30 21:49:16.669713 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.669593 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-config-volume\") pod \"dns-default-p8hw6\" (UID: \"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3\") " pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:16.669713 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.669629 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/35a08208-7245-4e1b-9c0b-9e825a808365-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-55pjz\" (UID: \"35a08208-7245-4e1b-9c0b-9e825a808365\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-55pjz" Oct 30 21:49:16.669713 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.669655 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgd6c\" (UniqueName: \"kubernetes.io/projected/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-kube-api-access-hgd6c\") pod \"dns-default-p8hw6\" (UID: \"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3\") " pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:16.669713 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.669675 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8258\" (UniqueName: \"kubernetes.io/projected/35a08208-7245-4e1b-9c0b-9e825a808365-kube-api-access-h8258\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-55pjz\" (UID: \"35a08208-7245-4e1b-9c0b-9e825a808365\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-55pjz" Oct 30 21:49:16.669713 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.669691 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-tmp-dir\") pod \"dns-default-p8hw6\" (UID: \"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3\") " pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:16.670019 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.669954 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-tmp-dir\") pod \"dns-default-p8hw6\" (UID: \"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3\") " pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:16.671015 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.670993 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11109f61-f4f0-4253-aacb-886c353a5772-config\") pod \"service-ca-operator-544d749c8c-88kh6\" (UID: \"11109f61-f4f0-4253-aacb-886c353a5772\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88kh6" Oct 30 21:49:16.671117 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:16.671065 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:49:16.671117 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.671060 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/35a08208-7245-4e1b-9c0b-9e825a808365-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-55pjz\" (UID: \"35a08208-7245-4e1b-9c0b-9e825a808365\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-55pjz" Oct 30 21:49:16.671212 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:16.671129 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-metrics-tls podName:3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:17.171116063 +0000 UTC m=+33.990849598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-metrics-tls") pod "dns-default-p8hw6" (UID: "3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3") : secret "dns-default-metrics-tls" not found Oct 30 21:49:16.671689 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.671659 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-config-volume\") pod \"dns-default-p8hw6\" (UID: \"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3\") " pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:16.673693 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.673672 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11109f61-f4f0-4253-aacb-886c353a5772-serving-cert\") pod \"service-ca-operator-544d749c8c-88kh6\" (UID: \"11109f61-f4f0-4253-aacb-886c353a5772\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88kh6" Oct 30 21:49:16.675141 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.675101 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/35a08208-7245-4e1b-9c0b-9e825a808365-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-55pjz\" (UID: \"35a08208-7245-4e1b-9c0b-9e825a808365\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-55pjz" Oct 30 21:49:16.732675 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.732635 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgccp\" (UniqueName: \"kubernetes.io/projected/11109f61-f4f0-4253-aacb-886c353a5772-kube-api-access-vgccp\") pod \"service-ca-operator-544d749c8c-88kh6\" (UID: \"11109f61-f4f0-4253-aacb-886c353a5772\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88kh6" Oct 30 21:49:16.733042 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.732799 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:49:16.733490 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.733468 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8258\" (UniqueName: \"kubernetes.io/projected/35a08208-7245-4e1b-9c0b-9e825a808365-kube-api-access-h8258\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-55pjz\" (UID: \"35a08208-7245-4e1b-9c0b-9e825a808365\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-55pjz" Oct 30 21:49:16.733585 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.733562 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67x9l\" (UniqueName: \"kubernetes.io/projected/226c41d5-de84-4950-978f-a1f0b4cbc319-kube-api-access-67x9l\") pod \"network-check-source-6f78d59f99-989z4\" (UID: \"226c41d5-de84-4950-978f-a1f0b4cbc319\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-989z4" Oct 30 21:49:16.734068 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.734043 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgd6c\" (UniqueName: \"kubernetes.io/projected/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-kube-api-access-hgd6c\") pod \"dns-default-p8hw6\" (UID: \"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3\") " pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:16.734291 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.734277 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8ctq\" (UniqueName: \"kubernetes.io/projected/4579f026-39bd-465c-b0db-41676370ba55-kube-api-access-l8ctq\") pod \"volume-data-source-validator-58c9f5f46d-p54hn\" (UID: \"4579f026-39bd-465c-b0db-41676370ba55\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p54hn" Oct 30 21:49:16.755425 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.755398 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-55pjz" Oct 30 21:49:16.770812 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.770789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn7gh\" (UniqueName: \"kubernetes.io/projected/c7fe5efb-7029-41b8-8bbe-f37f49410f27-kube-api-access-fn7gh\") pod \"ingress-canary-b6fbt\" (UID: \"c7fe5efb-7029-41b8-8bbe-f37f49410f27\") " pod="openshift-ingress-canary/ingress-canary-b6fbt" Oct 30 21:49:16.770893 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.770833 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7fe5efb-7029-41b8-8bbe-f37f49410f27-cert\") pod \"ingress-canary-b6fbt\" (UID: \"c7fe5efb-7029-41b8-8bbe-f37f49410f27\") " pod="openshift-ingress-canary/ingress-canary-b6fbt" Oct 30 21:49:16.773414 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.773389 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7fe5efb-7029-41b8-8bbe-f37f49410f27-cert\") pod \"ingress-canary-b6fbt\" (UID: \"c7fe5efb-7029-41b8-8bbe-f37f49410f27\") " pod="openshift-ingress-canary/ingress-canary-b6fbt" Oct 30 21:49:16.775124 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.775102 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p54hn" Oct 30 21:49:16.795083 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.795049 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88kh6" Oct 30 21:49:16.837344 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.837317 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-989z4" Oct 30 21:49:16.838400 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.838332 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn7gh\" (UniqueName: \"kubernetes.io/projected/c7fe5efb-7029-41b8-8bbe-f37f49410f27-kube-api-access-fn7gh\") pod \"ingress-canary-b6fbt\" (UID: \"c7fe5efb-7029-41b8-8bbe-f37f49410f27\") " pod="openshift-ingress-canary/ingress-canary-b6fbt" Oct 30 21:49:16.858141 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.857931 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b6fbt" Oct 30 21:49:16.936543 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.936509 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:16.973386 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:16.973359 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/47917159-29d7-45b9-bbf2-13cef4408de9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5xd2z\" (UID: \"47917159-29d7-45b9-bbf2-13cef4408de9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xd2z" Oct 30 21:49:16.973531 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:16.973476 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:49:16.973584 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:16.973532 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47917159-29d7-45b9-bbf2-13cef4408de9-networking-console-plugin-cert podName:47917159-29d7-45b9-bbf2-13cef4408de9 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:17.973518434 +0000 UTC m=+34.793251968 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/47917159-29d7-45b9-bbf2-13cef4408de9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5xd2z" (UID: "47917159-29d7-45b9-bbf2-13cef4408de9") : secret "networking-console-plugin-cert" not found Oct 30 21:49:17.010399 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.010261 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6pjx4"] Oct 30 21:49:17.012155 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.012020 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-k6p2v"] Oct 30 21:49:17.020332 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.020280 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2xctf"] Oct 30 21:49:17.037684 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:49:17.037649 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df43c88_a01f_4ec7_a385_d5afcf6f9e6c.slice/crio-f839a6af4cc59cc154537387101d77f74429e09b52a630505d1737c8a234134f WatchSource:0}: Error finding container f839a6af4cc59cc154537387101d77f74429e09b52a630505d1737c8a234134f: Status 404 returned error can't find the container with id f839a6af4cc59cc154537387101d77f74429e09b52a630505d1737c8a234134f Oct 30 21:49:17.074901 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.073916 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-w7pwq\" (UID: \"fc3ff956-6466-46e3-b734-13a3ac148d36\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq" Oct 30 21:49:17.074901 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.073969 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ba3d2b-2d12-458c-9984-1dd82c275e10-service-ca-bundle\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:17.074901 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.074003 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-metrics-certs\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:17.074901 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.074072 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-tls\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:17.074901 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.074096 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:17.074901 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.074154 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95097323-8907-45bc-8967-0de718adefbd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hmttl\" (UID: \"95097323-8907-45bc-8967-0de718adefbd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl" Oct 30 21:49:17.074901 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:17.074351 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:49:17.074901 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:17.074415 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95097323-8907-45bc-8967-0de718adefbd-cluster-monitoring-operator-tls podName:95097323-8907-45bc-8967-0de718adefbd nodeName:}" failed. No retries permitted until 2026-10-30 21:49:18.074396454 +0000 UTC m=+34.894130012 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/95097323-8907-45bc-8967-0de718adefbd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-hmttl" (UID: "95097323-8907-45bc-8967-0de718adefbd") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:49:17.074901 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:17.074480 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:49:17.074901 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:17.074515 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-tls podName:fc3ff956-6466-46e3-b734-13a3ac148d36 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:18.07450244 +0000 UTC m=+34.894235994 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-w7pwq" (UID: "fc3ff956-6466-46e3-b734-13a3ac148d36") : secret "samples-operator-tls" not found Oct 30 21:49:17.074901 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:17.074578 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/11ba3d2b-2d12-458c-9984-1dd82c275e10-service-ca-bundle podName:11ba3d2b-2d12-458c-9984-1dd82c275e10 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:18.074569498 +0000 UTC m=+34.894303032 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/11ba3d2b-2d12-458c-9984-1dd82c275e10-service-ca-bundle") pod "router-default-76684dbbf-8mwh7" (UID: "11ba3d2b-2d12-458c-9984-1dd82c275e10") : configmap references non-existent config key: service-ca.crt Oct 30 21:49:17.074901 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:17.074628 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:49:17.074901 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:17.074655 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-metrics-certs podName:11ba3d2b-2d12-458c-9984-1dd82c275e10 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:18.074646693 +0000 UTC m=+34.894380231 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-metrics-certs") pod "router-default-76684dbbf-8mwh7" (UID: "11ba3d2b-2d12-458c-9984-1dd82c275e10") : secret "router-metrics-certs-default" not found Oct 30 21:49:17.074901 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:17.074729 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:49:17.074901 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:17.074740 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68775ff9b8-rkd8f: secret "image-registry-tls" not found Oct 30 21:49:17.075734 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:17.074769 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-tls podName:de2247d3-e5ef-4a77-b087-8622bec117bc nodeName:}" failed. No retries permitted until 2026-10-30 21:49:18.074759496 +0000 UTC m=+34.894493050 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-tls") pod "image-registry-68775ff9b8-rkd8f" (UID: "de2247d3-e5ef-4a77-b087-8622bec117bc") : secret "image-registry-tls" not found Oct 30 21:49:17.075734 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:17.074821 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:49:17.075734 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:17.074830 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55fdbc6f6f-ngftj: secret "image-registry-tls" not found Oct 30 21:49:17.075734 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:17.074857 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls podName:84bfdb37-6346-4384-8d97-d43aba6f03fe nodeName:}" failed. No retries permitted until 2026-10-30 21:49:18.074847584 +0000 UTC m=+34.894581135 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls") pod "image-registry-55fdbc6f6f-ngftj" (UID: "84bfdb37-6346-4384-8d97-d43aba6f03fe") : secret "image-registry-tls" not found Oct 30 21:49:17.152997 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.152330 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-8w8b5"] Oct 30 21:49:17.153383 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.153315 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-55pjz"] Oct 30 21:49:17.159640 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.159613 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p54hn"] Oct 30 21:49:17.175580 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.175556 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-metrics-tls\") pod \"dns-default-p8hw6\" (UID: \"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3\") " pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:17.175734 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:17.175716 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:49:17.175801 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:17.175791 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-metrics-tls podName:3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:18.17576956 +0000 UTC m=+34.995503109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-metrics-tls") pod "dns-default-p8hw6" (UID: "3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3") : secret "dns-default-metrics-tls" not found Oct 30 21:49:17.184907 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:49:17.184874 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11109f61_f4f0_4253_aacb_886c353a5772.slice/crio-ab93116a47c2064a9a3779d9cae4a89403f0cc9d15d2431c9f8b04382b35c13f WatchSource:0}: Error finding container ab93116a47c2064a9a3779d9cae4a89403f0cc9d15d2431c9f8b04382b35c13f: Status 404 returned error can't find the container with id ab93116a47c2064a9a3779d9cae4a89403f0cc9d15d2431c9f8b04382b35c13f Oct 30 21:49:17.187133 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.186783 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-88kh6"] Oct 30 21:49:17.229168 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:49:17.229129 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226c41d5_de84_4950_978f_a1f0b4cbc319.slice/crio-eab77923780ae46182d22025f2db7c4e2214c1db453230aeee370578f48f89a2 WatchSource:0}: Error finding container eab77923780ae46182d22025f2db7c4e2214c1db453230aeee370578f48f89a2: Status 404 returned error can't find the container with id eab77923780ae46182d22025f2db7c4e2214c1db453230aeee370578f48f89a2 Oct 30 21:49:17.231078 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.231055 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-989z4"] Oct 30 21:49:17.271041 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:49:17.271011 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7fe5efb_7029_41b8_8bbe_f37f49410f27.slice/crio-c2fab8c8f94425955fda1227f35b73dad2362caa4ff1924c6bc2d05c5fd4f59e WatchSource:0}: Error finding container c2fab8c8f94425955fda1227f35b73dad2362caa4ff1924c6bc2d05c5fd4f59e: Status 404 returned error can't find the container with id c2fab8c8f94425955fda1227f35b73dad2362caa4ff1924c6bc2d05c5fd4f59e Oct 30 21:49:17.272821 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.272798 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b6fbt"] Oct 30 21:49:17.329662 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:49:17.329633 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0cece9c_2008_49ce_87de_4804d4e7de08.slice/crio-4f020130a1364da047db4e8b77083514ccee215eae989a164c46e413faef9d03 WatchSource:0}: Error finding container 4f020130a1364da047db4e8b77083514ccee215eae989a164c46e413faef9d03: Status 404 returned error can't find the container with id 4f020130a1364da047db4e8b77083514ccee215eae989a164c46e413faef9d03 Oct 30 21:49:17.331460 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.331416 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-jrhw4"] Oct 30 21:49:17.478641 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.478604 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs\") pod \"network-metrics-daemon-69j9v\" (UID: \"bd3b421a-9313-4f15-b81f-900717685fd6\") " pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:49:17.478830 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:17.478783 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:49:17.478889 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:17.478856 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs podName:bd3b421a-9313-4f15-b81f-900717685fd6 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:49.478836645 +0000 UTC m=+66.298570181 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs") pod "network-metrics-daemon-69j9v" (UID: "bd3b421a-9313-4f15-b81f-900717685fd6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:49:17.579241 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.579160 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret\") pod \"global-pull-secret-syncer-lfq7z\" (UID: \"1d430f7e-00f1-4b9e-9427-3e9971cfff31\") " pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:49:17.579411 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:17.579273 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:49:17.579411 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:17.579333 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret podName:1d430f7e-00f1-4b9e-9427-3e9971cfff31 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:49.579317547 +0000 UTC m=+66.399051086 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret") pod "global-pull-secret-syncer-lfq7z" (UID: "1d430f7e-00f1-4b9e-9427-3e9971cfff31") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:49:17.679892 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.679853 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk7l4\" (UniqueName: \"kubernetes.io/projected/550e690a-605e-47a4-ba95-74bcd26e0cc3-kube-api-access-kk7l4\") pod \"network-check-target-6gzkm\" (UID: \"550e690a-605e-47a4-ba95-74bcd26e0cc3\") " pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:49:17.682738 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.682708 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk7l4\" (UniqueName: \"kubernetes.io/projected/550e690a-605e-47a4-ba95-74bcd26e0cc3-kube-api-access-kk7l4\") pod \"network-check-target-6gzkm\" (UID: \"550e690a-605e-47a4-ba95-74bcd26e0cc3\") " pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:49:17.683653 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.683630 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:49:17.683750 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.683695 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:49:17.683750 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.683715 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:49:17.740535 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.740504 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:49:17.741227 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.741210 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9mgc8\"" Oct 30 21:49:17.741227 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.741220 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9bjlv\"" Oct 30 21:49:17.741339 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.741228 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jbflf\"" Oct 30 21:49:17.741339 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.741227 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:49:17.749522 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.749502 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:49:17.872774 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.872696 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-k6p2v" event={"ID":"9df43c88-a01f-4ec7-a385-d5afcf6f9e6c","Type":"ContainerStarted","Data":"ca8e8636c344df82881cb64fe8181106aa00c20bd61763e57a842cada104971a"} Oct 30 21:49:17.872912 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.872777 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-k6p2v" event={"ID":"9df43c88-a01f-4ec7-a385-d5afcf6f9e6c","Type":"ContainerStarted","Data":"f839a6af4cc59cc154537387101d77f74429e09b52a630505d1737c8a234134f"} Oct 30 21:49:17.873892 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.873866 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88kh6" event={"ID":"11109f61-f4f0-4253-aacb-886c353a5772","Type":"ContainerStarted","Data":"ab93116a47c2064a9a3779d9cae4a89403f0cc9d15d2431c9f8b04382b35c13f"} Oct 30 21:49:17.875196 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.875170 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-2xctf" event={"ID":"8bf0fd8f-2294-4729-9f49-a151de8627aa","Type":"ContainerStarted","Data":"9e82d6e7b83b5aa87e3a18c6a19a6af68af1a02969b9676fd2e33b845c053b0d"} Oct 30 21:49:17.875320 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.875210 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-2xctf" event={"ID":"8bf0fd8f-2294-4729-9f49-a151de8627aa","Type":"ContainerStarted","Data":"add64044c3798234efbd7b2232ce3f42b48efeb0e148bdbad067ad8b7b6ed9d3"} Oct 30 21:49:17.876421 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.876400 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p54hn" event={"ID":"4579f026-39bd-465c-b0db-41676370ba55","Type":"ContainerStarted","Data":"9f8402d180547bcff85e4b4c8359d241ae0dce587db164dda9246bdbf3ef7732"} Oct 30 21:49:17.877624 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.877598 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6pjx4" event={"ID":"0dce5b38-e1a7-41e3-8100-96af4e3c2b06","Type":"ContainerStarted","Data":"eb6a1b724b0e9040c1ea447597c7192a964e042342e1063364703dc8da5a8dd5"} Oct 30 21:49:17.877716 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.877633 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6pjx4" event={"ID":"0dce5b38-e1a7-41e3-8100-96af4e3c2b06","Type":"ContainerStarted","Data":"3d8709f5a2d9ef7a07b4bbf4c0ca1718a68539b27a9a989e532683c0467b8119"} Oct 30 21:49:17.881348 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.881326 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b6fbt" event={"ID":"c7fe5efb-7029-41b8-8bbe-f37f49410f27","Type":"ContainerStarted","Data":"c2fab8c8f94425955fda1227f35b73dad2362caa4ff1924c6bc2d05c5fd4f59e"} Oct 30 21:49:17.882521 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.882496 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-55pjz" event={"ID":"35a08208-7245-4e1b-9c0b-9e825a808365","Type":"ContainerStarted","Data":"49c17a497980cd740b0d8a74998b831e8d742944c245c732c35c7d2fafb86827"} Oct 30 21:49:17.883551 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.883531 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" event={"ID":"5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51","Type":"ContainerStarted","Data":"114ac894e52562dbb5f4c1eb70b3149ea1ecd5934c7693552db163ded70cd2c9"} Oct 30 21:49:17.884491 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.884474 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-jrhw4" event={"ID":"c0cece9c-2008-49ce-87de-4804d4e7de08","Type":"ContainerStarted","Data":"4f020130a1364da047db4e8b77083514ccee215eae989a164c46e413faef9d03"} Oct 30 21:49:17.885486 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.885467 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-989z4" event={"ID":"226c41d5-de84-4950-978f-a1f0b4cbc319","Type":"ContainerStarted","Data":"eab77923780ae46182d22025f2db7c4e2214c1db453230aeee370578f48f89a2"} Oct 30 21:49:17.982466 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:17.982413 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/47917159-29d7-45b9-bbf2-13cef4408de9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5xd2z\" (UID: \"47917159-29d7-45b9-bbf2-13cef4408de9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xd2z" Oct 30 21:49:17.982661 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:17.982613 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:49:17.982733 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:17.982686 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47917159-29d7-45b9-bbf2-13cef4408de9-networking-console-plugin-cert podName:47917159-29d7-45b9-bbf2-13cef4408de9 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:19.98266484 +0000 UTC m=+36.802398377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/47917159-29d7-45b9-bbf2-13cef4408de9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5xd2z" (UID: "47917159-29d7-45b9-bbf2-13cef4408de9") : secret "networking-console-plugin-cert" not found Oct 30 21:49:18.083818 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:18.083784 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-tls\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:18.083818 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:18.083819 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:18.084025 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:18.083921 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:49:18.084025 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:18.083927 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:49:18.084025 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:18.083946 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68775ff9b8-rkd8f: secret "image-registry-tls" not found Oct 30 21:49:18.084025 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:18.083963 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95097323-8907-45bc-8967-0de718adefbd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hmttl\" (UID: \"95097323-8907-45bc-8967-0de718adefbd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl" Oct 30 21:49:18.084025 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:18.083995 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-tls podName:de2247d3-e5ef-4a77-b087-8622bec117bc nodeName:}" failed. No retries permitted until 2026-10-30 21:49:20.083979983 +0000 UTC m=+36.903713521 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-tls") pod "image-registry-68775ff9b8-rkd8f" (UID: "de2247d3-e5ef-4a77-b087-8622bec117bc") : secret "image-registry-tls" not found Oct 30 21:49:18.084025 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:18.083935 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55fdbc6f6f-ngftj: secret "image-registry-tls" not found Oct 30 21:49:18.084374 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:18.084054 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:49:18.084374 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:18.084076 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls podName:84bfdb37-6346-4384-8d97-d43aba6f03fe nodeName:}" failed. No retries permitted until 2026-10-30 21:49:20.084060788 +0000 UTC m=+36.903794338 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls") pod "image-registry-55fdbc6f6f-ngftj" (UID: "84bfdb37-6346-4384-8d97-d43aba6f03fe") : secret "image-registry-tls" not found Oct 30 21:49:18.084374 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:18.084101 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-w7pwq\" (UID: \"fc3ff956-6466-46e3-b734-13a3ac148d36\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq" Oct 30 21:49:18.084374 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:18.084130 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ba3d2b-2d12-458c-9984-1dd82c275e10-service-ca-bundle\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:18.084374 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:18.084145 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95097323-8907-45bc-8967-0de718adefbd-cluster-monitoring-operator-tls podName:95097323-8907-45bc-8967-0de718adefbd nodeName:}" failed. No retries permitted until 2026-10-30 21:49:20.084130004 +0000 UTC m=+36.903863557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/95097323-8907-45bc-8967-0de718adefbd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-hmttl" (UID: "95097323-8907-45bc-8967-0de718adefbd") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:49:18.084374 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:18.084183 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-metrics-certs\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:18.084374 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:18.084188 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:49:18.084374 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:18.084223 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/11ba3d2b-2d12-458c-9984-1dd82c275e10-service-ca-bundle podName:11ba3d2b-2d12-458c-9984-1dd82c275e10 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:20.084212164 +0000 UTC m=+36.903945701 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/11ba3d2b-2d12-458c-9984-1dd82c275e10-service-ca-bundle") pod "router-default-76684dbbf-8mwh7" (UID: "11ba3d2b-2d12-458c-9984-1dd82c275e10") : configmap references non-existent config key: service-ca.crt Oct 30 21:49:18.084374 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:18.084224 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:49:18.084374 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:18.084269 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-tls podName:fc3ff956-6466-46e3-b734-13a3ac148d36 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:20.084242953 +0000 UTC m=+36.903976490 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-w7pwq" (UID: "fc3ff956-6466-46e3-b734-13a3ac148d36") : secret "samples-operator-tls" not found Oct 30 21:49:18.084374 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:18.084297 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-metrics-certs podName:11ba3d2b-2d12-458c-9984-1dd82c275e10 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:20.084285695 +0000 UTC m=+36.904019237 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-metrics-certs") pod "router-default-76684dbbf-8mwh7" (UID: "11ba3d2b-2d12-458c-9984-1dd82c275e10") : secret "router-metrics-certs-default" not found Oct 30 21:49:18.126721 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:18.126702 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6gzkm"] Oct 30 21:49:18.185272 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:18.185220 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-metrics-tls\") pod \"dns-default-p8hw6\" (UID: \"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3\") " pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:18.185436 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:18.185402 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:49:18.185510 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:18.185478 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-metrics-tls podName:3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:20.185457212 +0000 UTC m=+37.005190745 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-metrics-tls") pod "dns-default-p8hw6" (UID: "3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3") : secret "dns-default-metrics-tls" not found Oct 30 21:49:18.443656 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:18.443559 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-k6p2v" podStartSLOduration=405.443544544 podStartE2EDuration="6m45.443544544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:18.442872746 +0000 UTC m=+35.262606302" watchObservedRunningTime="2026-10-30 21:49:18.443544544 +0000 UTC m=+35.263278099" Oct 30 21:49:18.685353 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:18.685287 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-2xctf" podStartSLOduration=404.685265648 podStartE2EDuration="6m44.685265648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:18.564034305 +0000 UTC m=+35.383767862" watchObservedRunningTime="2026-10-30 21:49:18.685265648 +0000 UTC m=+35.504999197" Oct 30 21:49:18.898704 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:18.898397 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6gzkm" event={"ID":"550e690a-605e-47a4-ba95-74bcd26e0cc3","Type":"ContainerStarted","Data":"2c831fa197eb040ab59649744e4fa44b632eda251171ae2c0653ba688edf1edd"} Oct 30 21:49:20.004688 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:20.004639 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/47917159-29d7-45b9-bbf2-13cef4408de9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5xd2z\" (UID: \"47917159-29d7-45b9-bbf2-13cef4408de9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xd2z" Oct 30 21:49:20.005174 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:20.005036 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:49:20.005174 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:20.005101 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47917159-29d7-45b9-bbf2-13cef4408de9-networking-console-plugin-cert podName:47917159-29d7-45b9-bbf2-13cef4408de9 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:24.005087097 +0000 UTC m=+40.824820644 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/47917159-29d7-45b9-bbf2-13cef4408de9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5xd2z" (UID: "47917159-29d7-45b9-bbf2-13cef4408de9") : secret "networking-console-plugin-cert" not found Oct 30 21:49:20.105774 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:20.105735 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-w7pwq\" (UID: \"fc3ff956-6466-46e3-b734-13a3ac148d36\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq" Oct 30 21:49:20.105949 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:20.105785 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ba3d2b-2d12-458c-9984-1dd82c275e10-service-ca-bundle\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:20.105949 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:20.105802 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-metrics-certs\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:20.105949 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:20.105851 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-tls\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:20.105949 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:20.105867 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:20.105949 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:20.105903 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95097323-8907-45bc-8967-0de718adefbd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hmttl\" (UID: \"95097323-8907-45bc-8967-0de718adefbd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl" Oct 30 21:49:20.106321 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:20.106025 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:49:20.106321 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:20.106075 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95097323-8907-45bc-8967-0de718adefbd-cluster-monitoring-operator-tls podName:95097323-8907-45bc-8967-0de718adefbd nodeName:}" failed. No retries permitted until 2026-10-30 21:49:24.106062824 +0000 UTC m=+40.925796358 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/95097323-8907-45bc-8967-0de718adefbd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-hmttl" (UID: "95097323-8907-45bc-8967-0de718adefbd") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:49:20.106452 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:20.106408 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:49:20.106452 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:20.106448 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-tls podName:fc3ff956-6466-46e3-b734-13a3ac148d36 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:24.106436376 +0000 UTC m=+40.926169922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-w7pwq" (UID: "fc3ff956-6466-46e3-b734-13a3ac148d36") : secret "samples-operator-tls" not found Oct 30 21:49:20.106562 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:20.106493 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/11ba3d2b-2d12-458c-9984-1dd82c275e10-service-ca-bundle podName:11ba3d2b-2d12-458c-9984-1dd82c275e10 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:24.106486829 +0000 UTC m=+40.926220376 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/11ba3d2b-2d12-458c-9984-1dd82c275e10-service-ca-bundle") pod "router-default-76684dbbf-8mwh7" (UID: "11ba3d2b-2d12-458c-9984-1dd82c275e10") : configmap references non-existent config key: service-ca.crt Oct 30 21:49:20.106562 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:20.106529 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:49:20.106562 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:20.106549 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-metrics-certs podName:11ba3d2b-2d12-458c-9984-1dd82c275e10 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:24.106543829 +0000 UTC m=+40.926277376 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-metrics-certs") pod "router-default-76684dbbf-8mwh7" (UID: "11ba3d2b-2d12-458c-9984-1dd82c275e10") : secret "router-metrics-certs-default" not found Oct 30 21:49:20.106697 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:20.106590 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:49:20.106697 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:20.106597 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68775ff9b8-rkd8f: secret "image-registry-tls" not found Oct 30 21:49:20.106697 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:20.106616 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-tls podName:de2247d3-e5ef-4a77-b087-8622bec117bc nodeName:}" failed. No retries permitted until 2026-10-30 21:49:24.106609774 +0000 UTC m=+40.926343322 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-tls") pod "image-registry-68775ff9b8-rkd8f" (UID: "de2247d3-e5ef-4a77-b087-8622bec117bc") : secret "image-registry-tls" not found Oct 30 21:49:20.106697 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:20.106650 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:49:20.106697 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:20.106656 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55fdbc6f6f-ngftj: secret "image-registry-tls" not found Oct 30 21:49:20.106697 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:20.106672 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls podName:84bfdb37-6346-4384-8d97-d43aba6f03fe nodeName:}" failed. No retries permitted until 2026-10-30 21:49:24.106666847 +0000 UTC m=+40.926400394 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls") pod "image-registry-55fdbc6f6f-ngftj" (UID: "84bfdb37-6346-4384-8d97-d43aba6f03fe") : secret "image-registry-tls" not found Oct 30 21:49:20.206978 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:20.206679 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-metrics-tls\") pod \"dns-default-p8hw6\" (UID: \"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3\") " pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:20.206978 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:20.206855 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:49:20.206978 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:20.206933 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-metrics-tls podName:3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:24.206911851 +0000 UTC m=+41.026645403 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-metrics-tls") pod "dns-default-p8hw6" (UID: "3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3") : secret "dns-default-metrics-tls" not found Oct 30 21:49:20.915277 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:20.914802 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p54hn" event={"ID":"4579f026-39bd-465c-b0db-41676370ba55","Type":"ContainerStarted","Data":"fcb2ce9da5262356d23fc096a08a02125a9fb1f2afc5ce056a76719fcb10560a"} Oct 30 21:49:21.034497 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:21.034436 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6pjx4" podStartSLOduration=407.03441991 podStartE2EDuration="6m47.03441991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:18.684192806 +0000 UTC m=+35.503926364" watchObservedRunningTime="2026-10-30 21:49:21.03441991 +0000 UTC m=+37.854153465" Oct 30 21:49:21.035510 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:21.035467 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p54hn" podStartSLOduration=382.779576176 podStartE2EDuration="6m26.035454636s" totalImagesPullingTime="3.25587846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:21.033881848 +0000 UTC m=+37.853615405" watchObservedRunningTime="2026-10-30 21:49:21.035454636 +0000 UTC m=+37.855188192" Oct 30 21:49:22.925486 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:22.925395 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-55pjz" event={"ID":"35a08208-7245-4e1b-9c0b-9e825a808365","Type":"ContainerStarted","Data":"eb52ed220b3b8f921078ea044a235c08b4a3816a746a5ef554b68224eb7ad308"} Oct 30 21:49:22.928382 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:22.928342 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/0.log" Oct 30 21:49:22.928529 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:22.928383 2578 generic.go:356] "Generic (PLEG): container finished" podID="5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51" containerID="d7b840fac154f2112abd0491a6a57f5d620702810a4b4476acadbff80e5757a7" exitCode=255 Oct 30 21:49:22.928529 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:22.928448 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" event={"ID":"5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51","Type":"ContainerDied","Data":"d7b840fac154f2112abd0491a6a57f5d620702810a4b4476acadbff80e5757a7"} Oct 30 21:49:22.928820 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:22.928800 2578 scope.go:122] "RemoveContainer" containerID="d7b840fac154f2112abd0491a6a57f5d620702810a4b4476acadbff80e5757a7" Oct 30 21:49:23.046702 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:23.046637 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-55pjz" podStartSLOduration=383.579562967 podStartE2EDuration="6m29.046617522s" totalImagesPullingTime="5.467054555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:23.045217271 +0000 UTC m=+39.864950824" watchObservedRunningTime="2026-10-30 21:49:23.046617522 +0000 UTC m=+39.866351079" Oct 30 21:49:23.771889 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:23.771842 2578 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7fe5efb_7029_41b8_8bbe_f37f49410f27.slice/crio-a2080176050b930575f37602a1cf3cc6497e9d34065d12bd6dbd63d3813d77f6.scope\": RecentStats: unable to find data in memory cache]" Oct 30 21:49:23.934657 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:23.934498 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b6fbt" event={"ID":"c7fe5efb-7029-41b8-8bbe-f37f49410f27","Type":"ContainerStarted","Data":"a2080176050b930575f37602a1cf3cc6497e9d34065d12bd6dbd63d3813d77f6"} Oct 30 21:49:23.936498 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:23.936470 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 21:49:23.936999 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:23.936971 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/0.log" Oct 30 21:49:23.937112 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:23.937014 2578 generic.go:356] "Generic (PLEG): container finished" podID="5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51" containerID="b131ed16214c61457d42c061278f83843746a5a6736eb8f5a8d2ee4ddf210a2a" exitCode=255 Oct 30 21:49:23.937335 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:23.937309 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" event={"ID":"5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51","Type":"ContainerDied","Data":"b131ed16214c61457d42c061278f83843746a5a6736eb8f5a8d2ee4ddf210a2a"} Oct 30 21:49:23.937445 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:23.937370 2578 scope.go:122] "RemoveContainer" containerID="d7b840fac154f2112abd0491a6a57f5d620702810a4b4476acadbff80e5757a7" Oct 30 21:49:23.937445 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:23.937423 2578 scope.go:122] "RemoveContainer" containerID="b131ed16214c61457d42c061278f83843746a5a6736eb8f5a8d2ee4ddf210a2a" Oct 30 21:49:23.937848 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:23.937638 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-8w8b5_openshift-console-operator(5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" podUID="5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51" Oct 30 21:49:23.939042 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:23.939013 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-jrhw4" event={"ID":"c0cece9c-2008-49ce-87de-4804d4e7de08","Type":"ContainerStarted","Data":"56c0ca7e9b2bbdaa9faf5371a0665bd874ecbb21d852d6bf8f04f7ad410787ea"} Oct 30 21:49:23.940827 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:23.940790 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88kh6" event={"ID":"11109f61-f4f0-4253-aacb-886c353a5772","Type":"ContainerStarted","Data":"f1a4144efc243977f35c360dbc48feae88935bc5e232cf4b5fc9e02cffdb1059"} Oct 30 21:49:24.053570 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.053518 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/47917159-29d7-45b9-bbf2-13cef4408de9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5xd2z\" (UID: \"47917159-29d7-45b9-bbf2-13cef4408de9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xd2z" Oct 30 21:49:24.053784 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:24.053698 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:49:24.053784 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:24.053777 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47917159-29d7-45b9-bbf2-13cef4408de9-networking-console-plugin-cert podName:47917159-29d7-45b9-bbf2-13cef4408de9 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:32.053758674 +0000 UTC m=+48.873492210 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/47917159-29d7-45b9-bbf2-13cef4408de9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5xd2z" (UID: "47917159-29d7-45b9-bbf2-13cef4408de9") : secret "networking-console-plugin-cert" not found Oct 30 21:49:24.054789 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.054738 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b6fbt" podStartSLOduration=1.6352867340000001 podStartE2EDuration="8.054723255s" totalImagesPullingTime="6.419436521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:24.05376141 +0000 UTC m=+40.873494969" watchObservedRunningTime="2026-10-30 21:49:24.054723255 +0000 UTC m=+40.874456812" Oct 30 21:49:24.154577 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.154535 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-tls\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:24.154577 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.154580 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:24.154819 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.154636 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95097323-8907-45bc-8967-0de718adefbd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hmttl\" (UID: \"95097323-8907-45bc-8967-0de718adefbd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl" Oct 30 21:49:24.154819 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:24.154689 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:49:24.154819 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:24.154706 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68775ff9b8-rkd8f: secret "image-registry-tls" not found Oct 30 21:49:24.154819 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.154719 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-w7pwq\" (UID: \"fc3ff956-6466-46e3-b734-13a3ac148d36\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq" Oct 30 21:49:24.154819 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:24.154746 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:49:24.154819 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:24.154768 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-tls podName:de2247d3-e5ef-4a77-b087-8622bec117bc nodeName:}" failed. No retries permitted until 2026-10-30 21:49:32.154750614 +0000 UTC m=+48.974484165 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-tls") pod "image-registry-68775ff9b8-rkd8f" (UID: "de2247d3-e5ef-4a77-b087-8622bec117bc") : secret "image-registry-tls" not found Oct 30 21:49:24.154819 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:24.154767 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55fdbc6f6f-ngftj: secret "image-registry-tls" not found Oct 30 21:49:24.154819 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:24.154805 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls podName:84bfdb37-6346-4384-8d97-d43aba6f03fe nodeName:}" failed. No retries permitted until 2026-10-30 21:49:32.154795441 +0000 UTC m=+48.974528989 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls") pod "image-registry-55fdbc6f6f-ngftj" (UID: "84bfdb37-6346-4384-8d97-d43aba6f03fe") : secret "image-registry-tls" not found Oct 30 21:49:24.154819 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.154751 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ba3d2b-2d12-458c-9984-1dd82c275e10-service-ca-bundle\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:24.154819 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:24.154817 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:49:24.155326 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:24.154816 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:49:24.155326 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:24.154830 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/11ba3d2b-2d12-458c-9984-1dd82c275e10-service-ca-bundle podName:11ba3d2b-2d12-458c-9984-1dd82c275e10 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:32.154820112 +0000 UTC m=+48.974553658 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/11ba3d2b-2d12-458c-9984-1dd82c275e10-service-ca-bundle") pod "router-default-76684dbbf-8mwh7" (UID: "11ba3d2b-2d12-458c-9984-1dd82c275e10") : configmap references non-existent config key: service-ca.crt Oct 30 21:49:24.155326 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.154890 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-metrics-certs\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:24.155326 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:24.154958 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:49:24.155326 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:24.154993 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-metrics-certs podName:11ba3d2b-2d12-458c-9984-1dd82c275e10 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:32.154982653 +0000 UTC m=+48.974716192 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-metrics-certs") pod "router-default-76684dbbf-8mwh7" (UID: "11ba3d2b-2d12-458c-9984-1dd82c275e10") : secret "router-metrics-certs-default" not found Oct 30 21:49:24.155326 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:24.155009 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-tls podName:fc3ff956-6466-46e3-b734-13a3ac148d36 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:32.154997671 +0000 UTC m=+48.974731220 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-w7pwq" (UID: "fc3ff956-6466-46e3-b734-13a3ac148d36") : secret "samples-operator-tls" not found Oct 30 21:49:24.155326 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:24.155026 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95097323-8907-45bc-8967-0de718adefbd-cluster-monitoring-operator-tls podName:95097323-8907-45bc-8967-0de718adefbd nodeName:}" failed. No retries permitted until 2026-10-30 21:49:32.155017824 +0000 UTC m=+48.974751357 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/95097323-8907-45bc-8967-0de718adefbd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-hmttl" (UID: "95097323-8907-45bc-8967-0de718adefbd") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:49:24.173515 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.173465 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88kh6" podStartSLOduration=383.05405624 podStartE2EDuration="6m29.173450872s" totalImagesPullingTime="6.119394632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:24.17257551 +0000 UTC m=+40.992309067" watchObservedRunningTime="2026-10-30 21:49:24.173450872 +0000 UTC m=+40.993184489" Oct 30 21:49:24.256025 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.255946 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-metrics-tls\") pod \"dns-default-p8hw6\" (UID: \"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3\") " pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:24.256181 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:24.256076 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:49:24.256181 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:24.256140 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-metrics-tls podName:3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:32.256124121 +0000 UTC m=+49.075857673 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-metrics-tls") pod "dns-default-p8hw6" (UID: "3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3") : secret "dns-default-metrics-tls" not found Oct 30 21:49:24.294081 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.294013 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-jrhw4" podStartSLOduration=388.304092909 podStartE2EDuration="6m34.293996905s" totalImagesPullingTime="5.989903996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:24.293056065 +0000 UTC m=+41.112789621" watchObservedRunningTime="2026-10-30 21:49:24.293996905 +0000 UTC m=+41.113730517" Oct 30 21:49:24.806580 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.806546 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hkqxb"] Oct 30 21:49:24.824812 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.824780 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:24.862898 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.862861 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:24.863059 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.862910 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-826wq\" (UniqueName: \"kubernetes.io/projected/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-kube-api-access-826wq\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:24.863059 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.863036 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:24.863165 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.863065 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-27sld"] Oct 30 21:49:24.863165 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.863145 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-data-volume\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:24.863285 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.863233 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-crio-socket\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:24.881336 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.881301 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hkqxb"] Oct 30 21:49:24.881475 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.881391 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-f2dgh"] Oct 30 21:49:24.881524 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.881499 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-27sld" Oct 30 21:49:24.884289 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.884243 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:49:24.884436 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.884350 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bbh9g\"" Oct 30 21:49:24.885102 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.885087 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:49:24.896144 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.896120 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f2dgh" Oct 30 21:49:24.921285 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.921233 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-27sld"] Oct 30 21:49:24.934816 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.934781 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-f2dgh"] Oct 30 21:49:24.941877 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.941853 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 21:49:24.942183 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.942158 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 21:49:24.946402 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.946372 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6gzkm" event={"ID":"550e690a-605e-47a4-ba95-74bcd26e0cc3","Type":"ContainerStarted","Data":"11a18862f093c1f63c4e38acbe3091c5a5c4dd20ed20c3301c9369d8a97e7b3b"} Oct 30 21:49:24.946536 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.946433 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:49:24.948001 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.947984 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 21:49:24.948320 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.948302 2578 scope.go:122] "RemoveContainer" containerID="b131ed16214c61457d42c061278f83843746a5a6736eb8f5a8d2ee4ddf210a2a" Oct 30 21:49:24.948515 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:24.948491 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-8w8b5_openshift-console-operator(5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" podUID="5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51" Oct 30 21:49:24.949579 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.949557 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-989z4" event={"ID":"226c41d5-de84-4950-978f-a1f0b4cbc319","Type":"ContainerStarted","Data":"5a888b26d776c9ebdd81b0d4c49ff6a4cf964816e83e533a0a78264d69bfa909"} Oct 30 21:49:24.963879 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.963848 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdhdv\" (UniqueName: \"kubernetes.io/projected/f71e935d-f091-4f16-820b-8d7fc632f003-kube-api-access-vdhdv\") pod \"migrator-66df5d496b-f2dgh\" (UID: \"f71e935d-f091-4f16-820b-8d7fc632f003\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f2dgh" Oct 30 21:49:24.963879 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.963881 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f54bb\" (UniqueName: \"kubernetes.io/projected/6e73e56e-9619-4b97-bf20-a6291acf6355-kube-api-access-f54bb\") pod \"migrator-66df5d496b-27sld\" (UID: \"6e73e56e-9619-4b97-bf20-a6291acf6355\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-27sld" Oct 30 21:49:24.964881 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.964839 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:24.964986 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.964904 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-826wq\" (UniqueName: \"kubernetes.io/projected/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-kube-api-access-826wq\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:24.964986 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.964973 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:24.965276 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:24.965236 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:49:24.965401 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:24.965326 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-kube-rbac-proxy-cm podName:5b2dbc17-c725-4893-896e-4dd2ec2dbc13 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:25.4653047 +0000 UTC m=+42.285038237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hkqxb" (UID: "5b2dbc17-c725-4893-896e-4dd2ec2dbc13") : configmap "kube-rbac-proxy" not found Oct 30 21:49:24.965476 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:24.965429 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:49:24.965527 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:24.965484 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-insights-runtime-extractor-tls podName:5b2dbc17-c725-4893-896e-4dd2ec2dbc13 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:25.465469009 +0000 UTC m=+42.285202549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hkqxb" (UID: "5b2dbc17-c725-4893-896e-4dd2ec2dbc13") : secret "insights-runtime-extractor-tls" not found Oct 30 21:49:24.965593 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.965573 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-data-volume\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:24.965689 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.965672 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-crio-socket\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:24.965841 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.965806 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-crio-socket\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:24.967801 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.966742 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-data-volume\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:24.970673 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:24.970650 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-pf5gv\"" Oct 30 21:49:25.025647 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.025613 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-jr4s9-d8s6t"] Oct 30 21:49:25.032578 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.032546 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-826wq\" (UniqueName: \"kubernetes.io/projected/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-kube-api-access-826wq\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:25.047719 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.047697 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" Oct 30 21:49:25.066395 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.066314 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcm8z\" (UniqueName: \"kubernetes.io/projected/8b60900b-7a0e-4039-97af-c8157ae19ca0-kube-api-access-fcm8z\") pod \"periodic-gathering-jr4s9-d8s6t\" (UID: \"8b60900b-7a0e-4039-97af-c8157ae19ca0\") " pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" Oct 30 21:49:25.066395 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.066377 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b60900b-7a0e-4039-97af-c8157ae19ca0-service-ca-bundle\") pod \"periodic-gathering-jr4s9-d8s6t\" (UID: \"8b60900b-7a0e-4039-97af-c8157ae19ca0\") " pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" Oct 30 21:49:25.066577 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.066451 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdhdv\" (UniqueName: \"kubernetes.io/projected/f71e935d-f091-4f16-820b-8d7fc632f003-kube-api-access-vdhdv\") pod \"migrator-66df5d496b-f2dgh\" (UID: \"f71e935d-f091-4f16-820b-8d7fc632f003\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f2dgh" Oct 30 21:49:25.066577 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.066540 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f54bb\" (UniqueName: \"kubernetes.io/projected/6e73e56e-9619-4b97-bf20-a6291acf6355-kube-api-access-f54bb\") pod \"migrator-66df5d496b-27sld\" (UID: \"6e73e56e-9619-4b97-bf20-a6291acf6355\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-27sld" Oct 30 21:49:25.066695 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.066679 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8b60900b-7a0e-4039-97af-c8157ae19ca0-archives-path\") pod \"periodic-gathering-jr4s9-d8s6t\" (UID: \"8b60900b-7a0e-4039-97af-c8157ae19ca0\") " pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" Oct 30 21:49:25.066864 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.066845 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b60900b-7a0e-4039-97af-c8157ae19ca0-trusted-ca-bundle\") pod \"periodic-gathering-jr4s9-d8s6t\" (UID: \"8b60900b-7a0e-4039-97af-c8157ae19ca0\") " pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" Oct 30 21:49:25.080962 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.080914 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6gzkm" podStartSLOduration=34.873081986 podStartE2EDuration="41.080900015s" totalImagesPullingTime="6.207818029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:25.079359189 +0000 UTC m=+41.899092747" watchObservedRunningTime="2026-10-30 21:49:25.080900015 +0000 UTC m=+41.900633571" Oct 30 21:49:25.083787 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.083753 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-jr4s9-d8s6t"] Oct 30 21:49:25.129267 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.129217 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdhdv\" (UniqueName: \"kubernetes.io/projected/f71e935d-f091-4f16-820b-8d7fc632f003-kube-api-access-vdhdv\") pod \"migrator-66df5d496b-f2dgh\" (UID: \"f71e935d-f091-4f16-820b-8d7fc632f003\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f2dgh" Oct 30 21:49:25.129267 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.129223 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f54bb\" (UniqueName: \"kubernetes.io/projected/6e73e56e-9619-4b97-bf20-a6291acf6355-kube-api-access-f54bb\") pod \"migrator-66df5d496b-27sld\" (UID: \"6e73e56e-9619-4b97-bf20-a6291acf6355\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-27sld" Oct 30 21:49:25.167507 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.167471 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8b60900b-7a0e-4039-97af-c8157ae19ca0-archives-path\") pod \"periodic-gathering-jr4s9-d8s6t\" (UID: \"8b60900b-7a0e-4039-97af-c8157ae19ca0\") " pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" Oct 30 21:49:25.167670 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.167614 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b60900b-7a0e-4039-97af-c8157ae19ca0-trusted-ca-bundle\") pod \"periodic-gathering-jr4s9-d8s6t\" (UID: \"8b60900b-7a0e-4039-97af-c8157ae19ca0\") " pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" Oct 30 21:49:25.167736 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.167672 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcm8z\" (UniqueName: \"kubernetes.io/projected/8b60900b-7a0e-4039-97af-c8157ae19ca0-kube-api-access-fcm8z\") pod \"periodic-gathering-jr4s9-d8s6t\" (UID: \"8b60900b-7a0e-4039-97af-c8157ae19ca0\") " pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" Oct 30 21:49:25.167736 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.167730 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b60900b-7a0e-4039-97af-c8157ae19ca0-service-ca-bundle\") pod \"periodic-gathering-jr4s9-d8s6t\" (UID: \"8b60900b-7a0e-4039-97af-c8157ae19ca0\") " pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" Oct 30 21:49:25.167945 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.167924 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8b60900b-7a0e-4039-97af-c8157ae19ca0-archives-path\") pod \"periodic-gathering-jr4s9-d8s6t\" (UID: \"8b60900b-7a0e-4039-97af-c8157ae19ca0\") " pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" Oct 30 21:49:25.168295 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.168275 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b60900b-7a0e-4039-97af-c8157ae19ca0-service-ca-bundle\") pod \"periodic-gathering-jr4s9-d8s6t\" (UID: \"8b60900b-7a0e-4039-97af-c8157ae19ca0\") " pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" Oct 30 21:49:25.168641 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.168618 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b60900b-7a0e-4039-97af-c8157ae19ca0-trusted-ca-bundle\") pod \"periodic-gathering-jr4s9-d8s6t\" (UID: \"8b60900b-7a0e-4039-97af-c8157ae19ca0\") " pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" Oct 30 21:49:25.204017 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.203981 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-27sld" Oct 30 21:49:25.227193 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.227162 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f2dgh" Oct 30 21:49:25.231608 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.231574 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcm8z\" (UniqueName: \"kubernetes.io/projected/8b60900b-7a0e-4039-97af-c8157ae19ca0-kube-api-access-fcm8z\") pod \"periodic-gathering-jr4s9-d8s6t\" (UID: \"8b60900b-7a0e-4039-97af-c8157ae19ca0\") " pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" Oct 30 21:49:25.374392 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.374314 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" Oct 30 21:49:25.437632 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.437568 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-989z4" podStartSLOduration=356.344012425 podStartE2EDuration="6m3.437550216s" totalImagesPullingTime="7.093537791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:25.436336212 +0000 UTC m=+42.256069768" watchObservedRunningTime="2026-10-30 21:49:25.437550216 +0000 UTC m=+42.257283773" Oct 30 21:49:25.469863 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.469838 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:25.469958 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.469903 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:25.470008 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:25.469963 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:49:25.470055 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:25.470034 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-kube-rbac-proxy-cm podName:5b2dbc17-c725-4893-896e-4dd2ec2dbc13 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:26.47001759 +0000 UTC m=+43.289751127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hkqxb" (UID: "5b2dbc17-c725-4893-896e-4dd2ec2dbc13") : configmap "kube-rbac-proxy" not found Oct 30 21:49:25.470116 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:25.470078 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:49:25.470172 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:25.470149 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-insights-runtime-extractor-tls podName:5b2dbc17-c725-4893-896e-4dd2ec2dbc13 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:26.470131519 +0000 UTC m=+43.289865074 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hkqxb" (UID: "5b2dbc17-c725-4893-896e-4dd2ec2dbc13") : secret "insights-runtime-extractor-tls" not found Oct 30 21:49:25.594034 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:49:25.593992 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e73e56e_9619_4b97_bf20_a6291acf6355.slice/crio-373986040c020db99e3077377d47cc599f2eb297977682f180fd47a3c2e1c1f5 WatchSource:0}: Error finding container 373986040c020db99e3077377d47cc599f2eb297977682f180fd47a3c2e1c1f5: Status 404 returned error can't find the container with id 373986040c020db99e3077377d47cc599f2eb297977682f180fd47a3c2e1c1f5 Oct 30 21:49:25.597912 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.597882 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-27sld"] Oct 30 21:49:25.642057 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:49:25.642028 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71e935d_f091_4f16_820b_8d7fc632f003.slice/crio-c59a8eea64518cccb015decbdeff9327a62f4223b713bc4278344fd1041ce992 WatchSource:0}: Error finding container c59a8eea64518cccb015decbdeff9327a62f4223b713bc4278344fd1041ce992: Status 404 returned error can't find the container with id c59a8eea64518cccb015decbdeff9327a62f4223b713bc4278344fd1041ce992 Oct 30 21:49:25.644275 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.644220 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-f2dgh"] Oct 30 21:49:25.774297 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:49:25.774244 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b60900b_7a0e_4039_97af_c8157ae19ca0.slice/crio-9ea7a7f39436bd95eb87aaa921bf467e4ccb983ecc7b9cb726badd8d5ea2b22e WatchSource:0}: Error finding container 9ea7a7f39436bd95eb87aaa921bf467e4ccb983ecc7b9cb726badd8d5ea2b22e: Status 404 returned error can't find the container with id 9ea7a7f39436bd95eb87aaa921bf467e4ccb983ecc7b9cb726badd8d5ea2b22e Oct 30 21:49:25.776381 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.776362 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-jr4s9-d8s6t"] Oct 30 21:49:25.953764 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.953672 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-27sld" event={"ID":"6e73e56e-9619-4b97-bf20-a6291acf6355","Type":"ContainerStarted","Data":"373986040c020db99e3077377d47cc599f2eb297977682f180fd47a3c2e1c1f5"} Oct 30 21:49:25.955028 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.955001 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" event={"ID":"8b60900b-7a0e-4039-97af-c8157ae19ca0","Type":"ContainerStarted","Data":"c1443ed6373d5d19747e0f15060e8f266bfc59abf7fb41a8cc83e4c58ed79c16"} Oct 30 21:49:25.955154 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.955046 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" event={"ID":"8b60900b-7a0e-4039-97af-c8157ae19ca0","Type":"ContainerStarted","Data":"9ea7a7f39436bd95eb87aaa921bf467e4ccb983ecc7b9cb726badd8d5ea2b22e"} Oct 30 21:49:25.956119 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:25.956097 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f2dgh" event={"ID":"f71e935d-f091-4f16-820b-8d7fc632f003","Type":"ContainerStarted","Data":"c59a8eea64518cccb015decbdeff9327a62f4223b713bc4278344fd1041ce992"} Oct 30 21:49:26.073843 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.073790 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" podStartSLOduration=2.073775498 podStartE2EDuration="2.073775498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:26.073391045 +0000 UTC m=+42.893124597" watchObservedRunningTime="2026-10-30 21:49:26.073775498 +0000 UTC m=+42.893509054" Oct 30 21:49:26.078281 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.078241 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-mzhxr"] Oct 30 21:49:26.100023 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.099998 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-mzhxr" Oct 30 21:49:26.136669 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.136636 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-mzhxr"] Oct 30 21:49:26.153870 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.153842 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 21:49:26.154037 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.154022 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 21:49:26.154114 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.154048 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 21:49:26.154114 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.154051 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 21:49:26.154200 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.154149 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-tzzl2\"" Oct 30 21:49:26.176578 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.176548 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hnzg\" (UniqueName: \"kubernetes.io/projected/d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2-kube-api-access-9hnzg\") pod \"service-ca-5fffc97bdf-mzhxr\" (UID: \"d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mzhxr" Oct 30 21:49:26.176578 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.176581 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2-signing-cabundle\") pod \"service-ca-5fffc97bdf-mzhxr\" (UID: \"d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mzhxr" Oct 30 21:49:26.176759 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.176684 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2-signing-key\") pod \"service-ca-5fffc97bdf-mzhxr\" (UID: \"d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mzhxr" Oct 30 21:49:26.277712 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.277624 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hnzg\" (UniqueName: \"kubernetes.io/projected/d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2-kube-api-access-9hnzg\") pod \"service-ca-5fffc97bdf-mzhxr\" (UID: \"d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mzhxr" Oct 30 21:49:26.277712 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.277664 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2-signing-cabundle\") pod \"service-ca-5fffc97bdf-mzhxr\" (UID: \"d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mzhxr" Oct 30 21:49:26.277933 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.277744 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2-signing-key\") pod \"service-ca-5fffc97bdf-mzhxr\" (UID: \"d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mzhxr" Oct 30 21:49:26.278461 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.278437 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2-signing-cabundle\") pod \"service-ca-5fffc97bdf-mzhxr\" (UID: \"d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mzhxr" Oct 30 21:49:26.280159 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.280138 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2-signing-key\") pod \"service-ca-5fffc97bdf-mzhxr\" (UID: \"d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mzhxr" Oct 30 21:49:26.340532 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.340502 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hnzg\" (UniqueName: \"kubernetes.io/projected/d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2-kube-api-access-9hnzg\") pod \"service-ca-5fffc97bdf-mzhxr\" (UID: \"d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mzhxr" Oct 30 21:49:26.422952 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.422912 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-mzhxr" Oct 30 21:49:26.479357 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.479322 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:26.479500 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:26.479449 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:49:26.479540 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:26.479507 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-kube-rbac-proxy-cm podName:5b2dbc17-c725-4893-896e-4dd2ec2dbc13 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:28.479492659 +0000 UTC m=+45.299226223 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hkqxb" (UID: "5b2dbc17-c725-4893-896e-4dd2ec2dbc13") : configmap "kube-rbac-proxy" not found Oct 30 21:49:26.479540 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.479528 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:26.479692 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:26.479680 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:49:26.479726 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:26.479722 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-insights-runtime-extractor-tls podName:5b2dbc17-c725-4893-896e-4dd2ec2dbc13 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:28.479710255 +0000 UTC m=+45.299443806 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hkqxb" (UID: "5b2dbc17-c725-4893-896e-4dd2ec2dbc13") : secret "insights-runtime-extractor-tls" not found Oct 30 21:49:26.733458 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.733425 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:49:26.733599 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.733482 2578 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:49:26.733822 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.733809 2578 scope.go:122] "RemoveContainer" containerID="b131ed16214c61457d42c061278f83843746a5a6736eb8f5a8d2ee4ddf210a2a" Oct 30 21:49:26.734009 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:26.733988 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-8w8b5_openshift-console-operator(5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" podUID="5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51" Oct 30 21:49:26.831164 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:49:26.831131 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d7000a_8ca2_4dd3_9b87_a35afbfd2ad2.slice/crio-b6a2dd3b149a596b09c061b9b51c78112abf4aef1a98c011da199fb570c87008 WatchSource:0}: Error finding container b6a2dd3b149a596b09c061b9b51c78112abf4aef1a98c011da199fb570c87008: Status 404 returned error can't find the container with id b6a2dd3b149a596b09c061b9b51c78112abf4aef1a98c011da199fb570c87008 Oct 30 21:49:26.832084 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.832060 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-mzhxr"] Oct 30 21:49:26.974468 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.974328 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-mzhxr" event={"ID":"d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2","Type":"ContainerStarted","Data":"ea3093dc228b665c64086e28e13febca9e0c41fc308ac28e3be0997c3e0adaed"} Oct 30 21:49:26.974468 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:26.974420 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-mzhxr" event={"ID":"d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2","Type":"ContainerStarted","Data":"b6a2dd3b149a596b09c061b9b51c78112abf4aef1a98c011da199fb570c87008"} Oct 30 21:49:28.500439 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:28.500392 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:28.500938 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:28.500565 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:49:28.500938 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:28.500635 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-insights-runtime-extractor-tls podName:5b2dbc17-c725-4893-896e-4dd2ec2dbc13 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:32.500620274 +0000 UTC m=+49.320353824 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hkqxb" (UID: "5b2dbc17-c725-4893-896e-4dd2ec2dbc13") : secret "insights-runtime-extractor-tls" not found Oct 30 21:49:28.500938 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:28.500739 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:28.501342 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:28.501319 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:28.914222 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:28.914195 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-74zqz_1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4/dns-node-resolver/0.log" Oct 30 21:49:28.981874 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:28.981828 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f2dgh" event={"ID":"f71e935d-f091-4f16-820b-8d7fc632f003","Type":"ContainerStarted","Data":"35b7ed8f61bd0ffdad84f83787430da788a6f88e05a3b55c8e58c1c7ec67c1fd"} Oct 30 21:49:28.983336 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:28.983299 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-27sld" event={"ID":"6e73e56e-9619-4b97-bf20-a6291acf6355","Type":"ContainerStarted","Data":"b33b3d71bf29c901e5c56edc2af850c2b2dc2b287c546b2705099429938051ca"} Oct 30 21:49:29.103120 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:29.103069 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f2dgh" podStartSLOduration=2.714417012 podStartE2EDuration="5.103055398s" totalImagesPullingTime="2.388638386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:29.102645815 +0000 UTC m=+45.922379372" watchObservedRunningTime="2026-10-30 21:49:29.103055398 +0000 UTC m=+45.922788954" Oct 30 21:49:29.103848 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:29.103812 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-mzhxr" podStartSLOduration=3.103798628 podStartE2EDuration="3.103798628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:27.094485224 +0000 UTC m=+43.914218781" watchObservedRunningTime="2026-10-30 21:49:29.103798628 +0000 UTC m=+45.923532242" Oct 30 21:49:29.226779 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:29.226680 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-27sld" podStartSLOduration=2.791976649 podStartE2EDuration="5.226666419s" totalImagesPullingTime="2.43468977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:29.225404406 +0000 UTC m=+46.045137962" watchObservedRunningTime="2026-10-30 21:49:29.226666419 +0000 UTC m=+46.046399972" Oct 30 21:49:30.113850 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:30.113820 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x4655_ceedf44e-88c1-49cf-901e-75558703c361/node-ca/0.log" Oct 30 21:49:30.719868 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:30.719818 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b6fbt_c7fe5efb-7029-41b8-8bbe-f37f49410f27/serve-healthcheck-canary/0.log" Oct 30 21:49:30.914400 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:30.914372 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-27sld_6e73e56e-9619-4b97-bf20-a6291acf6355/migrator/0.log" Oct 30 21:49:31.120309 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:31.120278 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-f2dgh_f71e935d-f091-4f16-820b-8d7fc632f003/migrator/0.log" Oct 30 21:49:31.314950 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:31.314923 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-55pjz_35a08208-7245-4e1b-9c0b-9e825a808365/kube-storage-version-migrator-operator/0.log" Oct 30 21:49:32.137456 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:32.137414 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/47917159-29d7-45b9-bbf2-13cef4408de9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5xd2z\" (UID: \"47917159-29d7-45b9-bbf2-13cef4408de9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xd2z" Oct 30 21:49:32.139761 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:32.139731 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/47917159-29d7-45b9-bbf2-13cef4408de9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5xd2z\" (UID: \"47917159-29d7-45b9-bbf2-13cef4408de9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xd2z" Oct 30 21:49:32.198840 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:32.198802 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-5xd2z" Oct 30 21:49:32.238366 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:32.238332 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-tls\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:32.238366 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:32.238367 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:32.238555 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:32.238418 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95097323-8907-45bc-8967-0de718adefbd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hmttl\" (UID: \"95097323-8907-45bc-8967-0de718adefbd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl" Oct 30 21:49:32.238555 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:32.238480 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:49:32.238555 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:32.238493 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-w7pwq\" (UID: \"fc3ff956-6466-46e3-b734-13a3ac148d36\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq" Oct 30 21:49:32.238555 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:32.238500 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68775ff9b8-rkd8f: secret "image-registry-tls" not found Oct 30 21:49:32.238555 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:32.238527 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ba3d2b-2d12-458c-9984-1dd82c275e10-service-ca-bundle\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:32.238783 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:32.238565 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:49:32.238783 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:32.238581 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55fdbc6f6f-ngftj: secret "image-registry-tls" not found Oct 30 21:49:32.238783 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:32.238587 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-tls podName:de2247d3-e5ef-4a77-b087-8622bec117bc nodeName:}" failed. No retries permitted until 2026-10-30 21:49:48.2385672 +0000 UTC m=+65.058300750 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-tls") pod "image-registry-68775ff9b8-rkd8f" (UID: "de2247d3-e5ef-4a77-b087-8622bec117bc") : secret "image-registry-tls" not found Oct 30 21:49:32.238783 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:32.238621 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:49:32.238783 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:32.238627 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/11ba3d2b-2d12-458c-9984-1dd82c275e10-service-ca-bundle podName:11ba3d2b-2d12-458c-9984-1dd82c275e10 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:48.238613592 +0000 UTC m=+65.058347131 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/11ba3d2b-2d12-458c-9984-1dd82c275e10-service-ca-bundle") pod "router-default-76684dbbf-8mwh7" (UID: "11ba3d2b-2d12-458c-9984-1dd82c275e10") : configmap references non-existent config key: service-ca.crt Oct 30 21:49:32.238783 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:32.238680 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls podName:84bfdb37-6346-4384-8d97-d43aba6f03fe nodeName:}" failed. No retries permitted until 2026-10-30 21:49:48.238671749 +0000 UTC m=+65.058405283 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls") pod "image-registry-55fdbc6f6f-ngftj" (UID: "84bfdb37-6346-4384-8d97-d43aba6f03fe") : secret "image-registry-tls" not found Oct 30 21:49:32.238783 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:32.238699 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-metrics-certs\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:32.238783 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:32.238749 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:49:32.238783 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:32.238777 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-metrics-certs podName:11ba3d2b-2d12-458c-9984-1dd82c275e10 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:48.238769449 +0000 UTC m=+65.058502982 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-metrics-certs") pod "router-default-76684dbbf-8mwh7" (UID: "11ba3d2b-2d12-458c-9984-1dd82c275e10") : secret "router-metrics-certs-default" not found Oct 30 21:49:32.239146 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:32.238797 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95097323-8907-45bc-8967-0de718adefbd-cluster-monitoring-operator-tls podName:95097323-8907-45bc-8967-0de718adefbd nodeName:}" failed. No retries permitted until 2026-10-30 21:49:48.238782325 +0000 UTC m=+65.058515870 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/95097323-8907-45bc-8967-0de718adefbd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-hmttl" (UID: "95097323-8907-45bc-8967-0de718adefbd") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:49:32.240950 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:32.240926 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-w7pwq\" (UID: \"fc3ff956-6466-46e3-b734-13a3ac148d36\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq" Oct 30 21:49:32.339699 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:32.339660 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-metrics-tls\") pod \"dns-default-p8hw6\" (UID: \"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3\") " pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:32.342143 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:32.342114 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-metrics-tls\") pod \"dns-default-p8hw6\" (UID: \"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3\") " pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:32.479848 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:32.479755 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:32.523641 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:32.523607 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq" Oct 30 21:49:32.541856 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:32.541828 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:32.542027 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:32.541999 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:49:32.542131 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:32.542087 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-insights-runtime-extractor-tls podName:5b2dbc17-c725-4893-896e-4dd2ec2dbc13 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:40.542067126 +0000 UTC m=+57.361800675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hkqxb" (UID: "5b2dbc17-c725-4893-896e-4dd2ec2dbc13") : secret "insights-runtime-extractor-tls" not found Oct 30 21:49:32.567034 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:49:32.567000 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47917159_29d7_45b9_bbf2_13cef4408de9.slice/crio-dc529468b79e6b85d51b5854b713775940759b7a512c5c0ebacbe15b472060b1 WatchSource:0}: Error finding container dc529468b79e6b85d51b5854b713775940759b7a512c5c0ebacbe15b472060b1: Status 404 returned error can't find the container with id dc529468b79e6b85d51b5854b713775940759b7a512c5c0ebacbe15b472060b1 Oct 30 21:49:32.568982 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:32.568950 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-5xd2z"] Oct 30 21:49:32.847872 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:49:32.847842 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9cf4d3_e8aa_4a4b_aea3_88f4d2ab87c3.slice/crio-b1aee542906e797088cb6a6b0121f8a722ad89edfd3fc5a8bb6b9fd211dbd171 WatchSource:0}: Error finding container b1aee542906e797088cb6a6b0121f8a722ad89edfd3fc5a8bb6b9fd211dbd171: Status 404 returned error can't find the container with id b1aee542906e797088cb6a6b0121f8a722ad89edfd3fc5a8bb6b9fd211dbd171 Oct 30 21:49:32.849694 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:32.849673 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p8hw6"] Oct 30 21:49:32.899868 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:32.899839 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq"] Oct 30 21:49:32.998119 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:32.998086 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p8hw6" event={"ID":"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3","Type":"ContainerStarted","Data":"b1aee542906e797088cb6a6b0121f8a722ad89edfd3fc5a8bb6b9fd211dbd171"} Oct 30 21:49:32.999398 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:32.999370 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-5xd2z" event={"ID":"47917159-29d7-45b9-bbf2-13cef4408de9","Type":"ContainerStarted","Data":"dc529468b79e6b85d51b5854b713775940759b7a512c5c0ebacbe15b472060b1"} Oct 30 21:49:33.000542 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:33.000517 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq" event={"ID":"fc3ff956-6466-46e3-b734-13a3ac148d36","Type":"ContainerStarted","Data":"bbd26b51457e1453cc51ec773785a48957a5de44d7e58aaf6e47e64808180460"} Oct 30 21:49:35.011876 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:35.011834 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-5xd2z" event={"ID":"47917159-29d7-45b9-bbf2-13cef4408de9","Type":"ContainerStarted","Data":"d021926f502737c0bb0880926427c3251687b4e2faac3104b77f26d91b5cd76b"} Oct 30 21:49:35.131771 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:35.131709 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-86c478bb88-5xd2z" podStartSLOduration=353.357130548 podStartE2EDuration="5m55.131689876s" totalImagesPullingTime="1.774559328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:35.130072188 +0000 UTC m=+51.949805746" watchObservedRunningTime="2026-10-30 21:49:35.131689876 +0000 UTC m=+51.951423433" Oct 30 21:49:35.679419 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:35.679329 2578 scope.go:122] "RemoveContainer" containerID="b131ed16214c61457d42c061278f83843746a5a6736eb8f5a8d2ee4ddf210a2a" Oct 30 21:49:36.018060 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:36.017962 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p8hw6" event={"ID":"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3","Type":"ContainerStarted","Data":"341dac3d09b754c4b43620dc7151e4d0ee38cfe505b805e059920d5782a70cd5"} Oct 30 21:49:36.018060 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:36.018036 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:36.018569 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:36.018063 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p8hw6" event={"ID":"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3","Type":"ContainerStarted","Data":"9092df9abe3013ab9951a6e6a17ca335a546e75767ae6320329f3952d0e136d8"} Oct 30 21:49:36.019905 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:36.019880 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 21:49:36.020006 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:36.019977 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" event={"ID":"5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51","Type":"ContainerStarted","Data":"1142b9a276db19b0329d47ae5249a37a419a4af2d4d97bbbad42da88001e6fdd"} Oct 30 21:49:36.020233 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:36.020207 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:49:36.021629 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:36.021604 2578 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-8w8b5 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.21:8443/readyz\": dial tcp 10.132.0.21:8443: connect: connection refused" start-of-body= Oct 30 21:49:36.021742 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:36.021646 2578 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" podUID="5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.21:8443/readyz\": dial tcp 10.132.0.21:8443: connect: connection refused" Oct 30 21:49:36.021944 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:36.021920 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq" event={"ID":"fc3ff956-6466-46e3-b734-13a3ac148d36","Type":"ContainerStarted","Data":"1b39e198b8a46366429dcd7ae16e20b217d07d63ab42cde1f230f98ab302405a"} Oct 30 21:49:36.022039 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:36.021971 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq" event={"ID":"fc3ff956-6466-46e3-b734-13a3ac148d36","Type":"ContainerStarted","Data":"e5a2e91bc4f8757903799678effbc6bf9b1506b94c619ff4413c78be239e340a"} Oct 30 21:49:36.142402 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:36.142349 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-p8hw6" podStartSLOduration=17.645492561 podStartE2EDuration="20.142334025s" totalImagesPullingTime="2.496841464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:36.141614827 +0000 UTC m=+52.961348384" watchObservedRunningTime="2026-10-30 21:49:36.142334025 +0000 UTC m=+52.962067580" Oct 30 21:49:36.263875 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:36.263819 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq" podStartSLOduration=405.805712277 podStartE2EDuration="6m48.263806782s" totalImagesPullingTime="2.458094505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:36.262810419 +0000 UTC m=+53.082543975" watchObservedRunningTime="2026-10-30 21:49:36.263806782 +0000 UTC m=+53.083540379" Oct 30 21:49:36.384197 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:36.384145 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" podStartSLOduration=396.927236028 podStartE2EDuration="6m42.384131004s" totalImagesPullingTime="5.456894976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:36.383759708 +0000 UTC m=+53.203493264" watchObservedRunningTime="2026-10-30 21:49:36.384131004 +0000 UTC m=+53.203864557" Oct 30 21:49:37.031055 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:37.031025 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:49:40.618581 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:40.618539 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:40.620970 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:40.620943 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:40.776351 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:40.776308 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:40.866084 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:40.865761 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:41.167944 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:41.167908 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hkqxb"] Oct 30 21:49:41.176408 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:49:41.176380 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2dbc17_c725_4893_896e_4dd2ec2dbc13.slice/crio-180790b45601d52b4ff7416e7ab9c05ee09a770df969fc05917948b9dde6f85d WatchSource:0}: Error finding container 180790b45601d52b4ff7416e7ab9c05ee09a770df969fc05917948b9dde6f85d: Status 404 returned error can't find the container with id 180790b45601d52b4ff7416e7ab9c05ee09a770df969fc05917948b9dde6f85d Oct 30 21:49:42.050968 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:42.050936 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hkqxb" event={"ID":"5b2dbc17-c725-4893-896e-4dd2ec2dbc13","Type":"ContainerStarted","Data":"ff766e1e5eb92ca4842a35dacb5b23b37026e5aea6690a51dfcc110db43a9fe4"} Oct 30 21:49:42.051380 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:42.050990 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hkqxb" event={"ID":"5b2dbc17-c725-4893-896e-4dd2ec2dbc13","Type":"ContainerStarted","Data":"180790b45601d52b4ff7416e7ab9c05ee09a770df969fc05917948b9dde6f85d"} Oct 30 21:49:43.055631 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:43.055590 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hkqxb" event={"ID":"5b2dbc17-c725-4893-896e-4dd2ec2dbc13","Type":"ContainerStarted","Data":"8e1ed29231f61442b61ede964642adb1f2d6f5a428a8e32937474d56749ccb19"} Oct 30 21:49:45.065519 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:45.065484 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hkqxb" event={"ID":"5b2dbc17-c725-4893-896e-4dd2ec2dbc13","Type":"ContainerStarted","Data":"44e4727bfd335e4bbeecfb935b4535925e96ce3e523fdbb24969e25c4592f218"} Oct 30 21:49:45.187538 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:45.187437 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hkqxb" podStartSLOduration=17.65216399 podStartE2EDuration="21.187424262s" totalImagesPullingTime="3.535260272s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:45.186317529 +0000 UTC m=+62.006051086" watchObservedRunningTime="2026-10-30 21:49:45.187424262 +0000 UTC m=+62.007157818" Oct 30 21:49:46.028935 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:46.028899 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:48.080893 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.080856 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-tqt4b"] Oct 30 21:49:48.084266 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.084228 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-tqt4b" Oct 30 21:49:48.103141 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.103102 2578 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68775ff9b8-rkd8f"] Oct 30 21:49:48.103388 ip-10-0-135-7 kubenswrapper[2578]: E1030 21:49:48.103366 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" podUID="de2247d3-e5ef-4a77-b087-8622bec117bc" Oct 30 21:49:48.138960 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.138924 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 21:49:48.139221 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.139204 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-nrl58\"" Oct 30 21:49:48.143424 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.143404 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 21:49:48.182838 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.182801 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c68f8f5c6-wt2pc"] Oct 30 21:49:48.186777 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.186748 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:48.188273 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.188218 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5gt6\" (UniqueName: \"kubernetes.io/projected/ce930258-60d9-4988-913c-2f3cb78eac8d-kube-api-access-v5gt6\") pod \"downloads-6dd789c8b7-tqt4b\" (UID: \"ce930258-60d9-4988-913c-2f3cb78eac8d\") " pod="openshift-console/downloads-6dd789c8b7-tqt4b" Oct 30 21:49:48.188390 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.188365 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce930258-60d9-4988-913c-2f3cb78eac8d-tmp\") pod \"downloads-6dd789c8b7-tqt4b\" (UID: \"ce930258-60d9-4988-913c-2f3cb78eac8d\") " pod="openshift-console/downloads-6dd789c8b7-tqt4b" Oct 30 21:49:48.190919 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.190892 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-tqt4b"] Oct 30 21:49:48.249016 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.248982 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c68f8f5c6-wt2pc"] Oct 30 21:49:48.289614 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.289578 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95097323-8907-45bc-8967-0de718adefbd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hmttl\" (UID: \"95097323-8907-45bc-8967-0de718adefbd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl" Oct 30 21:49:48.289767 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.289633 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5gt6\" (UniqueName: \"kubernetes.io/projected/ce930258-60d9-4988-913c-2f3cb78eac8d-kube-api-access-v5gt6\") pod \"downloads-6dd789c8b7-tqt4b\" (UID: \"ce930258-60d9-4988-913c-2f3cb78eac8d\") " pod="openshift-console/downloads-6dd789c8b7-tqt4b" Oct 30 21:49:48.289767 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.289668 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3458688b-8142-492c-ab8b-bffec754c8c8-image-registry-private-configuration\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:48.289767 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.289698 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce930258-60d9-4988-913c-2f3cb78eac8d-tmp\") pod \"downloads-6dd789c8b7-tqt4b\" (UID: \"ce930258-60d9-4988-913c-2f3cb78eac8d\") " pod="openshift-console/downloads-6dd789c8b7-tqt4b" Oct 30 21:49:48.289767 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.289724 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ba3d2b-2d12-458c-9984-1dd82c275e10-service-ca-bundle\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:48.289767 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.289748 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-metrics-certs\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:48.290018 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.289796 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3458688b-8142-492c-ab8b-bffec754c8c8-ca-trust-extracted\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:48.290018 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.289834 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3458688b-8142-492c-ab8b-bffec754c8c8-registry-certificates\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:48.290018 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.289997 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3458688b-8142-492c-ab8b-bffec754c8c8-trusted-ca\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:48.290155 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.290060 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-tls\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:48.290155 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.290109 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:48.290155 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.290135 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce930258-60d9-4988-913c-2f3cb78eac8d-tmp\") pod \"downloads-6dd789c8b7-tqt4b\" (UID: \"ce930258-60d9-4988-913c-2f3cb78eac8d\") " pod="openshift-console/downloads-6dd789c8b7-tqt4b" Oct 30 21:49:48.290155 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.290146 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3458688b-8142-492c-ab8b-bffec754c8c8-registry-tls\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:48.290372 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.290170 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxnj9\" (UniqueName: \"kubernetes.io/projected/3458688b-8142-492c-ab8b-bffec754c8c8-kube-api-access-hxnj9\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:48.290372 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.290266 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3458688b-8142-492c-ab8b-bffec754c8c8-installation-pull-secrets\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:48.290372 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.290322 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3458688b-8142-492c-ab8b-bffec754c8c8-bound-sa-token\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:48.290545 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.290515 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ba3d2b-2d12-458c-9984-1dd82c275e10-service-ca-bundle\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:48.292329 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.292307 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95097323-8907-45bc-8967-0de718adefbd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hmttl\" (UID: \"95097323-8907-45bc-8967-0de718adefbd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl" Oct 30 21:49:48.292553 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.292529 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:48.292671 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.292597 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-tls\") pod \"image-registry-68775ff9b8-rkd8f\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:48.292923 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.292904 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-metrics-certs\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:48.357344 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.357267 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5gt6\" (UniqueName: \"kubernetes.io/projected/ce930258-60d9-4988-913c-2f3cb78eac8d-kube-api-access-v5gt6\") pod \"downloads-6dd789c8b7-tqt4b\" (UID: \"ce930258-60d9-4988-913c-2f3cb78eac8d\") " pod="openshift-console/downloads-6dd789c8b7-tqt4b" Oct 30 21:49:48.391206 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.391161 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3458688b-8142-492c-ab8b-bffec754c8c8-registry-tls\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:48.391206 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.391201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxnj9\" (UniqueName: \"kubernetes.io/projected/3458688b-8142-492c-ab8b-bffec754c8c8-kube-api-access-hxnj9\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:48.391458 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.391230 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3458688b-8142-492c-ab8b-bffec754c8c8-installation-pull-secrets\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:48.391458 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.391245 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3458688b-8142-492c-ab8b-bffec754c8c8-bound-sa-token\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:48.391532 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.391510 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3458688b-8142-492c-ab8b-bffec754c8c8-image-registry-private-configuration\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:48.391619 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.391600 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3458688b-8142-492c-ab8b-bffec754c8c8-ca-trust-extracted\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:48.391677 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.391643 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3458688b-8142-492c-ab8b-bffec754c8c8-registry-certificates\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:48.391730 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.391689 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3458688b-8142-492c-ab8b-bffec754c8c8-trusted-ca\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:48.392102 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.392072 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3458688b-8142-492c-ab8b-bffec754c8c8-ca-trust-extracted\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:48.392553 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.392532 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3458688b-8142-492c-ab8b-bffec754c8c8-registry-certificates\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:48.393067 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.393038 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3458688b-8142-492c-ab8b-bffec754c8c8-trusted-ca\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:48.393851 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.393829 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3458688b-8142-492c-ab8b-bffec754c8c8-registry-tls\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:48.394001 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.393982 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3458688b-8142-492c-ab8b-bffec754c8c8-installation-pull-secrets\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:48.394060 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.394019 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3458688b-8142-492c-ab8b-bffec754c8c8-image-registry-private-configuration\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:48.395675 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.395660 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-tqt4b" Oct 30 21:49:48.462788 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.462752 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3458688b-8142-492c-ab8b-bffec754c8c8-bound-sa-token\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:48.465302 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.465273 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxnj9\" (UniqueName: \"kubernetes.io/projected/3458688b-8142-492c-ab8b-bffec754c8c8-kube-api-access-hxnj9\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:48.503451 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.503413 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-4wbzb\"" Oct 30 21:49:48.511325 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.511286 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl" Oct 30 21:49:48.524904 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.524880 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8rvzh\"" Oct 30 21:49:48.529978 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.529947 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:48.532235 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.532216 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:48.564299 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.564231 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-7fn9v\"" Oct 30 21:49:48.572816 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.572473 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:48.813871 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:49:48.813845 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce930258_60d9_4988_913c_2f3cb78eac8d.slice/crio-72df4ed43b9eb340b736a6f8f63ed88621c05e05d2c0630828120e93b3bac8c2 WatchSource:0}: Error finding container 72df4ed43b9eb340b736a6f8f63ed88621c05e05d2c0630828120e93b3bac8c2: Status 404 returned error can't find the container with id 72df4ed43b9eb340b736a6f8f63ed88621c05e05d2c0630828120e93b3bac8c2 Oct 30 21:49:48.815513 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.815440 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-tqt4b"] Oct 30 21:49:48.907332 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:49:48.907297 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95097323_8907_45bc_8967_0de718adefbd.slice/crio-e3fff7abd1c2d54ab14f33d63025e973a308ca7bb52976feceb7d4eab6d7ba1c WatchSource:0}: Error finding container e3fff7abd1c2d54ab14f33d63025e973a308ca7bb52976feceb7d4eab6d7ba1c: Status 404 returned error can't find the container with id e3fff7abd1c2d54ab14f33d63025e973a308ca7bb52976feceb7d4eab6d7ba1c Oct 30 21:49:48.909500 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.909479 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl"] Oct 30 21:49:48.957214 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:49:48.957180 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3458688b_8142_492c_ab8b_bffec754c8c8.slice/crio-7140c65db18f03b4c55457261bcf97ff25e415df1bc664655ff2f0d85916a07e WatchSource:0}: Error finding container 7140c65db18f03b4c55457261bcf97ff25e415df1bc664655ff2f0d85916a07e: Status 404 returned error can't find the container with id 7140c65db18f03b4c55457261bcf97ff25e415df1bc664655ff2f0d85916a07e Oct 30 21:49:48.959028 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.959008 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c68f8f5c6-wt2pc"] Oct 30 21:49:48.966504 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:49:48.966478 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84bfdb37_6346_4384_8d97_d43aba6f03fe.slice/crio-4d9c6673201363e110fb9941032ebf8c83fc19e1fe99fe2539d7b22f07db02c6 WatchSource:0}: Error finding container 4d9c6673201363e110fb9941032ebf8c83fc19e1fe99fe2539d7b22f07db02c6: Status 404 returned error can't find the container with id 4d9c6673201363e110fb9941032ebf8c83fc19e1fe99fe2539d7b22f07db02c6 Oct 30 21:49:48.967439 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.967414 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55fdbc6f6f-ngftj"] Oct 30 21:49:48.980173 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:48.980149 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-76684dbbf-8mwh7"] Oct 30 21:49:48.989302 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:49:48.989280 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ba3d2b_2d12_458c_9984_1dd82c275e10.slice/crio-379969793eebc3c4a8ee7b787852642f2d0bab68b5e4b5c5ca7277ebc1584eb7 WatchSource:0}: Error finding container 379969793eebc3c4a8ee7b787852642f2d0bab68b5e4b5c5ca7277ebc1584eb7: Status 404 returned error can't find the container with id 379969793eebc3c4a8ee7b787852642f2d0bab68b5e4b5c5ca7277ebc1584eb7 Oct 30 21:49:49.082082 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.082047 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-76684dbbf-8mwh7" event={"ID":"11ba3d2b-2d12-458c-9984-1dd82c275e10","Type":"ContainerStarted","Data":"24c4f899b9beaecaa430fe3f227b07d1f7cd3f52409d6c14ccf2c9853700e0ac"} Oct 30 21:49:49.082497 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.082106 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-76684dbbf-8mwh7" event={"ID":"11ba3d2b-2d12-458c-9984-1dd82c275e10","Type":"ContainerStarted","Data":"379969793eebc3c4a8ee7b787852642f2d0bab68b5e4b5c5ca7277ebc1584eb7"} Oct 30 21:49:49.084049 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.084015 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" event={"ID":"84bfdb37-6346-4384-8d97-d43aba6f03fe","Type":"ContainerStarted","Data":"0fe6220e95f2245e721938f428ebed95acae6b241b6f700a621c9c69e676f8be"} Oct 30 21:49:49.084168 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.084071 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" event={"ID":"84bfdb37-6346-4384-8d97-d43aba6f03fe","Type":"ContainerStarted","Data":"4d9c6673201363e110fb9941032ebf8c83fc19e1fe99fe2539d7b22f07db02c6"} Oct 30 21:49:49.084381 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.084363 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:49.085645 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.085610 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-tqt4b" event={"ID":"ce930258-60d9-4988-913c-2f3cb78eac8d","Type":"ContainerStarted","Data":"72df4ed43b9eb340b736a6f8f63ed88621c05e05d2c0630828120e93b3bac8c2"} Oct 30 21:49:49.087301 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.087276 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" event={"ID":"3458688b-8142-492c-ab8b-bffec754c8c8","Type":"ContainerStarted","Data":"9f9a2e898ab5978c7b2dad60acb00ad6f9f2b9e75fc23f1aef92f2262f634ca0"} Oct 30 21:49:49.087427 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.087317 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" event={"ID":"3458688b-8142-492c-ab8b-bffec754c8c8","Type":"ContainerStarted","Data":"7140c65db18f03b4c55457261bcf97ff25e415df1bc664655ff2f0d85916a07e"} Oct 30 21:49:49.087427 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.087356 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:49.088453 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.088428 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl" event={"ID":"95097323-8907-45bc-8967-0de718adefbd","Type":"ContainerStarted","Data":"e3fff7abd1c2d54ab14f33d63025e973a308ca7bb52976feceb7d4eab6d7ba1c"} Oct 30 21:49:49.088453 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.088448 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:49.095776 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.095745 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:49.200820 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.200729 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de2247d3-e5ef-4a77-b087-8622bec117bc-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de2247d3-e5ef-4a77-b087-8622bec117bc-trusted-ca\") pod \"de2247d3-e5ef-4a77-b087-8622bec117bc\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " Oct 30 21:49:49.200820 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.200808 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de2247d3-e5ef-4a77-b087-8622bec117bc-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/de2247d3-e5ef-4a77-b087-8622bec117bc-image-registry-private-configuration\") pod \"de2247d3-e5ef-4a77-b087-8622bec117bc\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " Oct 30 21:49:49.201057 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.200921 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-kube-api-access-8p75b\" (UniqueName: \"kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-kube-api-access-8p75b\") pod \"de2247d3-e5ef-4a77-b087-8622bec117bc\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " Oct 30 21:49:49.201057 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.200984 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-certificates\") pod \"de2247d3-e5ef-4a77-b087-8622bec117bc\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " Oct 30 21:49:49.201057 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.201023 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-bound-sa-token\") pod \"de2247d3-e5ef-4a77-b087-8622bec117bc\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " Oct 30 21:49:49.201211 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.201062 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de2247d3-e5ef-4a77-b087-8622bec117bc-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/de2247d3-e5ef-4a77-b087-8622bec117bc-ca-trust-extracted\") pod \"de2247d3-e5ef-4a77-b087-8622bec117bc\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " Oct 30 21:49:49.201211 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.201130 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de2247d3-e5ef-4a77-b087-8622bec117bc-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/de2247d3-e5ef-4a77-b087-8622bec117bc-installation-pull-secrets\") pod \"de2247d3-e5ef-4a77-b087-8622bec117bc\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " Oct 30 21:49:49.201211 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.201172 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-tls\" (UniqueName: \"kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-tls\") pod \"de2247d3-e5ef-4a77-b087-8622bec117bc\" (UID: \"de2247d3-e5ef-4a77-b087-8622bec117bc\") " Oct 30 21:49:49.201400 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.201214 2578 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de2247d3-e5ef-4a77-b087-8622bec117bc-trusted-ca" pod "de2247d3-e5ef-4a77-b087-8622bec117bc" (UID: "de2247d3-e5ef-4a77-b087-8622bec117bc"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:49.201400 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.201379 2578 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2247d3-e5ef-4a77-b087-8622bec117bc-ca-trust-extracted" pod "de2247d3-e5ef-4a77-b087-8622bec117bc" (UID: "de2247d3-e5ef-4a77-b087-8622bec117bc"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:49.201504 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.201379 2578 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-certificates" pod "de2247d3-e5ef-4a77-b087-8622bec117bc" (UID: "de2247d3-e5ef-4a77-b087-8622bec117bc"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:49.208290 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.207022 2578 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-tls" pod "de2247d3-e5ef-4a77-b087-8622bec117bc" (UID: "de2247d3-e5ef-4a77-b087-8622bec117bc"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:49.209289 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.208977 2578 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-kube-api-access-8p75b" pod "de2247d3-e5ef-4a77-b087-8622bec117bc" (UID: "de2247d3-e5ef-4a77-b087-8622bec117bc"). InnerVolumeSpecName "kube-api-access-8p75b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:49.209289 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.209169 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de2247d3-e5ef-4a77-b087-8622bec117bc-trusted-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:49.209289 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.209193 2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-certificates\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:49.209289 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.209209 2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/de2247d3-e5ef-4a77-b087-8622bec117bc-ca-trust-extracted\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:49.211728 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.211694 2578 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de2247d3-e5ef-4a77-b087-8622bec117bc-image-registry-private-configuration" pod "de2247d3-e5ef-4a77-b087-8622bec117bc" (UID: "de2247d3-e5ef-4a77-b087-8622bec117bc"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:49.211868 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.211810 2578 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de2247d3-e5ef-4a77-b087-8622bec117bc-installation-pull-secrets" pod "de2247d3-e5ef-4a77-b087-8622bec117bc" (UID: "de2247d3-e5ef-4a77-b087-8622bec117bc"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:49.212374 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.212333 2578 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-bound-sa-token" pod "de2247d3-e5ef-4a77-b087-8622bec117bc" (UID: "de2247d3-e5ef-4a77-b087-8622bec117bc"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:49.310674 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.310640 2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/de2247d3-e5ef-4a77-b087-8622bec117bc-installation-pull-secrets\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:49.310674 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.310677 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-registry-tls\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:49.310674 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.310693 2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/de2247d3-e5ef-4a77-b087-8622bec117bc-image-registry-private-configuration\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:49.310983 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.310713 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p75b\" (UniqueName: \"kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-kube-api-access-8p75b\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:49.310983 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.310727 2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de2247d3-e5ef-4a77-b087-8622bec117bc-bound-sa-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:49.337549 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.337479 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/router-default-76684dbbf-8mwh7" podStartSLOduration=415.337456826 podStartE2EDuration="6m55.337456826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:49.212450384 +0000 UTC m=+66.032183941" watchObservedRunningTime="2026-10-30 21:49:49.337456826 +0000 UTC m=+66.157190383" Oct 30 21:49:49.337778 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.337753 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" podStartSLOduration=1.337746701 podStartE2EDuration="1.337746701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:49.335707818 +0000 UTC m=+66.155441394" watchObservedRunningTime="2026-10-30 21:49:49.337746701 +0000 UTC m=+66.157480256" Oct 30 21:49:49.464113 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.464002 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" podStartSLOduration=56.463988037 podStartE2EDuration="56.463988037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:49.462779791 +0000 UTC m=+66.282513347" watchObservedRunningTime="2026-10-30 21:49:49.463988037 +0000 UTC m=+66.283721593" Oct 30 21:49:49.512922 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.512876 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs\") pod \"network-metrics-daemon-69j9v\" (UID: \"bd3b421a-9313-4f15-b81f-900717685fd6\") " pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:49:49.566102 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.566071 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:49:49.573409 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.573379 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:49.576817 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.576787 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs\") pod \"network-metrics-daemon-69j9v\" (UID: \"bd3b421a-9313-4f15-b81f-900717685fd6\") " pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:49:49.577019 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.577004 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:49.613848 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.613808 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret\") pod \"global-pull-secret-syncer-lfq7z\" (UID: \"1d430f7e-00f1-4b9e-9427-3e9971cfff31\") " pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:49:49.667044 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.667017 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:49:49.676526 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.676495 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret\") pod \"global-pull-secret-syncer-lfq7z\" (UID: \"1d430f7e-00f1-4b9e-9427-3e9971cfff31\") " pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:49:49.857161 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.857124 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jbflf\"" Oct 30 21:49:49.865237 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.865205 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:49:49.887024 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.886993 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9bjlv\"" Oct 30 21:49:49.895085 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:49.895052 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:49:50.093721 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:50.093617 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68775ff9b8-rkd8f" Oct 30 21:49:50.095076 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:50.094616 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:50.095707 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:50.095683 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:50.264228 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:50.264198 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lfq7z"] Oct 30 21:49:50.264726 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:49:50.264695 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d430f7e_00f1_4b9e_9427_3e9971cfff31.slice/crio-d6c5d5133bfcf2acea021dfa7d0287a4297dbe1a1f03d2bb38e27b73b5ac71ef WatchSource:0}: Error finding container d6c5d5133bfcf2acea021dfa7d0287a4297dbe1a1f03d2bb38e27b73b5ac71ef: Status 404 returned error can't find the container with id d6c5d5133bfcf2acea021dfa7d0287a4297dbe1a1f03d2bb38e27b73b5ac71ef Oct 30 21:49:50.291817 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:49:50.291780 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3b421a_9313_4f15_b81f_900717685fd6.slice/crio-2942f9ed8024d9ebefc206e92519f5b963850505ad500f668cc5035fdf6e7edf WatchSource:0}: Error finding container 2942f9ed8024d9ebefc206e92519f5b963850505ad500f668cc5035fdf6e7edf: Status 404 returned error can't find the container with id 2942f9ed8024d9ebefc206e92519f5b963850505ad500f668cc5035fdf6e7edf Oct 30 21:49:50.298296 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:50.291783 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-69j9v"] Oct 30 21:49:50.337000 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:50.336925 2578 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68775ff9b8-rkd8f"] Oct 30 21:49:50.348835 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:50.348801 2578 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-68775ff9b8-rkd8f"] Oct 30 21:49:51.100425 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:51.100384 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-69j9v" event={"ID":"bd3b421a-9313-4f15-b81f-900717685fd6","Type":"ContainerStarted","Data":"2942f9ed8024d9ebefc206e92519f5b963850505ad500f668cc5035fdf6e7edf"} Oct 30 21:49:51.102148 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:51.102117 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lfq7z" event={"ID":"1d430f7e-00f1-4b9e-9427-3e9971cfff31","Type":"ContainerStarted","Data":"d6c5d5133bfcf2acea021dfa7d0287a4297dbe1a1f03d2bb38e27b73b5ac71ef"} Oct 30 21:49:51.686757 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:51.686672 2578 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de2247d3-e5ef-4a77-b087-8622bec117bc" path="/var/lib/kubelet/pods/de2247d3-e5ef-4a77-b087-8622bec117bc/volumes" Oct 30 21:49:52.116408 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:52.116368 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl" event={"ID":"95097323-8907-45bc-8967-0de718adefbd","Type":"ContainerStarted","Data":"d463cdc9fddfadfdd56acfd6b4432e2ce8c33123e051aa95c8a90b0baf7b2acd"} Oct 30 21:49:52.308140 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:52.308077 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl" podStartSLOduration=420.878468465 podStartE2EDuration="7m3.308061202s" totalImagesPullingTime="2.429592737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:52.305881689 +0000 UTC m=+69.125615245" watchObservedRunningTime="2026-10-30 21:49:52.308061202 +0000 UTC m=+69.127794811" Oct 30 21:49:53.122281 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:53.122169 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-69j9v" event={"ID":"bd3b421a-9313-4f15-b81f-900717685fd6","Type":"ContainerStarted","Data":"13b852de1e6a7252355866aa359c8f940c53dc88554f72cd2bde0d47c64e4608"} Oct 30 21:49:53.122281 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:53.122218 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-69j9v" event={"ID":"bd3b421a-9313-4f15-b81f-900717685fd6","Type":"ContainerStarted","Data":"37c27217e399d7f78c23ab5654d7ad3c9b0ce03a2f97bcd4b65f2eadc023d808"} Oct 30 21:49:53.242766 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:53.242704 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-69j9v" podStartSLOduration=67.507988873 podStartE2EDuration="1m9.24268692s" totalImagesPullingTime="1.734698047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:53.242657482 +0000 UTC m=+70.062391040" watchObservedRunningTime="2026-10-30 21:49:53.24268692 +0000 UTC m=+70.062420478" Oct 30 21:49:53.828693 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:53.828654 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bc452"] Oct 30 21:49:53.832505 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:53.832477 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bc452" Oct 30 21:49:53.886790 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:53.886750 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 21:49:53.886972 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:53.886907 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-zp2pt\"" Oct 30 21:49:53.887354 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:53.887329 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bc452"] Oct 30 21:49:53.958572 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:53.958474 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8df7e7fd-be92-4d37-af33-930b7b36cbe6-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-bc452\" (UID: \"8df7e7fd-be92-4d37-af33-930b7b36cbe6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bc452" Oct 30 21:49:54.059574 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:54.059536 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8df7e7fd-be92-4d37-af33-930b7b36cbe6-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-bc452\" (UID: \"8df7e7fd-be92-4d37-af33-930b7b36cbe6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bc452" Oct 30 21:49:54.062674 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:54.062647 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8df7e7fd-be92-4d37-af33-930b7b36cbe6-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-bc452\" (UID: \"8df7e7fd-be92-4d37-af33-930b7b36cbe6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bc452" Oct 30 21:49:54.147232 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:54.146956 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bc452" Oct 30 21:49:54.549026 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:54.548975 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bc452"] Oct 30 21:49:54.557433 ip-10-0-135-7 kubenswrapper[2578]: W1030 21:49:54.557396 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df7e7fd_be92_4d37_af33_930b7b36cbe6.slice/crio-25c043a82f378ee1f768d77bc37682bee28ed6f94b374c58924c70d296371cda WatchSource:0}: Error finding container 25c043a82f378ee1f768d77bc37682bee28ed6f94b374c58924c70d296371cda: Status 404 returned error can't find the container with id 25c043a82f378ee1f768d77bc37682bee28ed6f94b374c58924c70d296371cda Oct 30 21:49:55.139830 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:55.139752 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bc452" event={"ID":"8df7e7fd-be92-4d37-af33-930b7b36cbe6","Type":"ContainerStarted","Data":"25c043a82f378ee1f768d77bc37682bee28ed6f94b374c58924c70d296371cda"} Oct 30 21:49:55.959438 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:55.958500 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:49:57.155545 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:57.153001 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bc452" event={"ID":"8df7e7fd-be92-4d37-af33-930b7b36cbe6","Type":"ContainerStarted","Data":"ba486bbe15581cd365563f6cd1ec44fd7d1833fdeb73aa86a0c17667b76097b8"} Oct 30 21:49:57.155545 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:57.154205 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bc452" Oct 30 21:49:57.157486 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:57.157453 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lfq7z" event={"ID":"1d430f7e-00f1-4b9e-9427-3e9971cfff31","Type":"ContainerStarted","Data":"27177fdbde857288551692294b86c1daf8383bb66fbc1d55c566d5588c798f72"} Oct 30 21:49:57.160416 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:57.160391 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bc452" Oct 30 21:49:57.272634 ip-10-0-135-7 kubenswrapper[2578]: I1030 21:49:57.272522 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bc452" podStartSLOduration=2.185950224 podStartE2EDuration="4.272501571s" totalImagesPullingTime="2.086551347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:57.271469609 +0000 UTC m=+74.091203168" watchObservedRunningTime="2026-10-30 21:49:57.272501571 +0000 UTC m=+74.092235129" Oct 30 21:49:57.389301 ip-10-0-135-7 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 21:49:57.400001 ip-10-0-135-7 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 21:49:57.400307 ip-10-0-135-7 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 21:49:57.400369 ip-10-0-135-7 systemd[1]: kubelet.service: Consumed 5.045s CPU time. Oct 30 21:49:57.412062 ip-10-0-135-7 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:49:57.497756 ip-10-0-135-7 kubenswrapper[6006]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:49:57.497756 ip-10-0-135-7 kubenswrapper[6006]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:49:57.497756 ip-10-0-135-7 kubenswrapper[6006]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:49:57.497756 ip-10-0-135-7 kubenswrapper[6006]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:49:57.499973 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.499883 6006 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:49:57.499973 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.499975 6006 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:49:57.500055 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.499981 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:49:57.500055 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.499985 6006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:49:57.500055 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.499989 6006 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:49:57.500055 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.499993 6006 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:49:57.500055 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.499998 6006 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:49:57.500055 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500002 6006 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:49:57.500055 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500007 6006 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:49:57.500055 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500012 6006 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:49:57.500055 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500017 6006 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:49:57.500055 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500021 6006 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:49:57.500055 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500026 6006 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:49:57.500055 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500029 6006 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:49:57.500055 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500032 6006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:49:57.500055 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500035 6006 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:49:57.500055 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500038 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:49:57.500055 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500040 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:49:57.500055 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500043 6006 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:49:57.500055 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500046 6006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:49:57.500055 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500048 6006 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:49:57.500055 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500051 6006 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:49:57.500603 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500054 6006 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:49:57.500603 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500057 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:49:57.500603 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500060 6006 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:49:57.500603 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500063 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:49:57.500603 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500066 6006 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:49:57.500603 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500069 6006 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:49:57.500603 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500071 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:49:57.500603 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500074 6006 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:49:57.500603 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500077 6006 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:49:57.500603 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500079 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:49:57.500603 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500082 6006 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:49:57.500603 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500085 6006 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:49:57.500603 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500087 6006 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:49:57.500603 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500091 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:49:57.500603 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500094 6006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:49:57.500603 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500096 6006 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:49:57.500603 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500099 6006 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:49:57.500603 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500102 6006 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:49:57.500603 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500104 6006 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:49:57.501094 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500107 6006 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:49:57.501094 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500110 6006 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:49:57.501094 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500112 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:49:57.501094 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500115 6006 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:49:57.501094 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500117 6006 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:49:57.501094 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500121 6006 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:49:57.501094 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500124 6006 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:49:57.501094 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500133 6006 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:49:57.501094 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500137 6006 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:49:57.501094 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500139 6006 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:49:57.501094 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500144 6006 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:49:57.501094 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500147 6006 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:49:57.501094 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500151 6006 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:49:57.501094 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500154 6006 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:49:57.501094 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500156 6006 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:49:57.501094 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500159 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:49:57.501094 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500162 6006 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:49:57.501094 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500164 6006 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:49:57.501094 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500167 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:49:57.501094 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500170 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:49:57.501625 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500172 6006 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:49:57.501625 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500175 6006 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:49:57.501625 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500177 6006 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:49:57.501625 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500180 6006 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:49:57.501625 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500183 6006 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:49:57.501625 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500187 6006 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:49:57.501625 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500189 6006 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:49:57.501625 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500192 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:49:57.501625 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500195 6006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:49:57.501625 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500197 6006 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:49:57.501625 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500200 6006 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:49:57.501625 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500203 6006 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:49:57.501625 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500205 6006 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:49:57.501625 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500208 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:49:57.501625 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500211 6006 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:49:57.501625 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500213 6006 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:49:57.501625 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500216 6006 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:49:57.501625 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500219 6006 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:49:57.501625 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500222 6006 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:49:57.501625 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500224 6006 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:49:57.502131 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500227 6006 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:49:57.502131 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500229 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:49:57.502131 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500232 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:49:57.502131 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500235 6006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:49:57.502131 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500238 6006 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:49:57.502131 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500240 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:49:57.502131 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500243 6006 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:49:57.502131 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500257 6006 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:49:57.502131 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500260 6006 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:49:57.502131 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500263 6006 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:49:57.502131 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500266 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:49:57.502131 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500268 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:49:57.502131 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500271 6006 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:49:57.502131 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500274 6006 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:49:57.502131 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500278 6006 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:49:57.502131 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500280 6006 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:49:57.502131 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500283 6006 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:49:57.502131 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500287 6006 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:49:57.502131 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500290 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:49:57.502616 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500293 6006 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:49:57.502616 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500297 6006 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:49:57.502616 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500300 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:49:57.502616 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500304 6006 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:49:57.502616 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500306 6006 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:49:57.502616 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.500309 6006 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:49:57.502616 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501068 6006 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:49:57.502616 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501074 6006 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:49:57.502616 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501077 6006 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:49:57.502616 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501080 6006 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:49:57.502616 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501084 6006 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:49:57.502616 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501087 6006 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:49:57.502616 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501090 6006 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:49:57.502616 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501093 6006 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:49:57.502616 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501096 6006 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:49:57.502616 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501099 6006 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:49:57.502616 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501102 6006 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:49:57.502616 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501105 6006 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:49:57.502616 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501107 6006 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:49:57.502616 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501110 6006 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:49:57.503136 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501112 6006 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:49:57.503136 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501115 6006 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:49:57.503136 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501118 6006 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:49:57.503136 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501120 6006 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:49:57.503136 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501123 6006 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:49:57.503136 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501126 6006 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:49:57.503136 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501128 6006 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:49:57.503136 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501132 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:49:57.503136 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501134 6006 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:49:57.503136 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501137 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:49:57.503136 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501140 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:49:57.503136 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501143 6006 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:49:57.503136 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501146 6006 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:49:57.503136 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501148 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:49:57.503136 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501151 6006 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:49:57.503136 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501154 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:49:57.503136 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501156 6006 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:49:57.503136 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501159 6006 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:49:57.503136 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501163 6006 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:49:57.503630 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501167 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:49:57.503630 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501170 6006 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:49:57.503630 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501173 6006 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:49:57.503630 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501176 6006 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:49:57.503630 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501179 6006 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:49:57.503630 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501181 6006 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:49:57.503630 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501184 6006 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:49:57.503630 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501187 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:49:57.503630 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501189 6006 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:49:57.503630 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501192 6006 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:49:57.503630 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501195 6006 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:49:57.503630 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501198 6006 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:49:57.503630 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501200 6006 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:49:57.503630 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501203 6006 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:49:57.503630 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501205 6006 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:49:57.503630 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501208 6006 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:49:57.503630 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501215 6006 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:49:57.503630 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501218 6006 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:49:57.503630 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501221 6006 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:49:57.503630 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501224 6006 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:49:57.504184 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501226 6006 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:49:57.504184 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501229 6006 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:49:57.504184 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501232 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:49:57.504184 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501235 6006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:49:57.504184 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501238 6006 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:49:57.504184 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501241 6006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:49:57.504184 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501244 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:49:57.504184 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501260 6006 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:49:57.504184 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501263 6006 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:49:57.504184 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501266 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:49:57.504184 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501269 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:49:57.504184 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501271 6006 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:49:57.504184 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501275 6006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:49:57.504184 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501278 6006 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:49:57.504184 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501280 6006 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:49:57.504184 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501283 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:49:57.504184 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501286 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:49:57.504184 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501288 6006 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:49:57.504184 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501292 6006 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:49:57.504690 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501295 6006 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:49:57.504690 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501297 6006 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:49:57.504690 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501300 6006 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:49:57.504690 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501303 6006 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:49:57.504690 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501305 6006 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:49:57.504690 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501308 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:49:57.504690 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501311 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:49:57.504690 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501313 6006 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:49:57.504690 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501316 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:49:57.504690 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501319 6006 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:49:57.504690 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501321 6006 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:49:57.504690 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501324 6006 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:49:57.504690 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501327 6006 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:49:57.504690 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501330 6006 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:49:57.504690 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501333 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:49:57.504690 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501338 6006 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:49:57.504690 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501341 6006 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:49:57.504690 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501344 6006 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:49:57.504690 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501346 6006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:49:57.504690 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501355 6006 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:49:57.505189 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501358 6006 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:49:57.505189 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501361 6006 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:49:57.505189 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501363 6006 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:49:57.505189 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501366 6006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:49:57.505189 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501368 6006 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:49:57.505189 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501373 6006 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:49:57.505189 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501377 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:49:57.505189 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501380 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:49:57.505189 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501382 6006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:49:57.505189 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501385 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:49:57.505189 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501388 6006 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:49:57.505189 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501392 6006 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:49:57.505189 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501394 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:49:57.505189 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501397 6006 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:49:57.505189 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501938 6006 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:49:57.505189 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501944 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:49:57.505189 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501947 6006 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:49:57.505189 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501950 6006 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:49:57.505189 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501953 6006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:49:57.505189 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501956 6006 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:49:57.505722 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501958 6006 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:49:57.505722 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501961 6006 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:49:57.505722 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501964 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:49:57.505722 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501967 6006 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:49:57.505722 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501969 6006 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:49:57.505722 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501972 6006 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:49:57.505722 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501975 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:49:57.505722 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501978 6006 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:49:57.505722 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501981 6006 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:49:57.505722 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501984 6006 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:49:57.505722 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501987 6006 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:49:57.505722 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501993 6006 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:49:57.505722 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501996 6006 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:49:57.505722 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.501998 6006 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:49:57.505722 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502003 6006 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:49:57.505722 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502007 6006 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:49:57.505722 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502011 6006 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:49:57.505722 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502014 6006 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:49:57.505722 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502017 6006 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:49:57.506441 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502020 6006 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:49:57.506441 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502023 6006 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:49:57.506441 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502025 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:49:57.506441 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502028 6006 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:49:57.506441 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502030 6006 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:49:57.506441 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502033 6006 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:49:57.506441 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502036 6006 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:49:57.506441 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502038 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:49:57.506441 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502041 6006 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:49:57.506441 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502044 6006 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:49:57.506441 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502047 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:49:57.506441 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502049 6006 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:49:57.506441 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502052 6006 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:49:57.506441 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502055 6006 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:49:57.506441 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502057 6006 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:49:57.506441 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502060 6006 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:49:57.506441 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502063 6006 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:49:57.506441 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502066 6006 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:49:57.506441 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502069 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:49:57.507234 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502071 6006 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:49:57.507234 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502074 6006 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:49:57.507234 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502077 6006 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:49:57.507234 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502079 6006 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:49:57.507234 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502083 6006 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:49:57.507234 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502087 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:49:57.507234 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502097 6006 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:49:57.507234 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502104 6006 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:49:57.507234 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502108 6006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:49:57.507234 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502113 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:49:57.507234 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502117 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:49:57.507234 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502121 6006 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:49:57.507234 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502124 6006 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:49:57.507234 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502126 6006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:49:57.507234 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502129 6006 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:49:57.507234 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502132 6006 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:49:57.507234 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502135 6006 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:49:57.507234 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502138 6006 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:49:57.507234 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502141 6006 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:49:57.508004 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502144 6006 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:49:57.508004 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502147 6006 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:49:57.508004 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502149 6006 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:49:57.508004 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502152 6006 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:49:57.508004 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502155 6006 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:49:57.508004 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502157 6006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:49:57.508004 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502160 6006 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:49:57.508004 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502163 6006 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:49:57.508004 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502166 6006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:49:57.508004 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502169 6006 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:49:57.508004 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502171 6006 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:49:57.508004 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502174 6006 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:49:57.508004 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502176 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:49:57.508004 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502179 6006 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:49:57.508004 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502182 6006 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:49:57.508004 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502195 6006 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:49:57.508004 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502198 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:49:57.508004 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502201 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:49:57.508004 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502203 6006 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:49:57.508004 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502206 6006 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:49:57.508553 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502214 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:49:57.508553 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502217 6006 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:49:57.508553 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502220 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:49:57.508553 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502223 6006 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:49:57.508553 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502225 6006 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:49:57.508553 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502228 6006 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:49:57.508553 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502232 6006 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:49:57.508553 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502236 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:49:57.508553 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502239 6006 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:49:57.508553 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502242 6006 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:49:57.508553 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502262 6006 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:49:57.508553 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502266 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:49:57.508553 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502269 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:49:57.508553 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502271 6006 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:49:57.508553 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502274 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:49:57.508553 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502277 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:49:57.508553 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502279 6006 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:49:57.508553 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502282 6006 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:49:57.508553 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502287 6006 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:49:57.509032 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502289 6006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:49:57.509032 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502292 6006 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:49:57.509032 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502295 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:49:57.509032 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.502297 6006 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:49:57.509032 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502391 6006 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:49:57.509032 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502406 6006 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:49:57.509032 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502413 6006 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:49:57.509032 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502418 6006 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:49:57.509032 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502422 6006 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:49:57.509032 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502426 6006 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:49:57.509032 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502430 6006 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:49:57.509032 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502435 6006 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:49:57.509032 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502439 6006 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:49:57.509032 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502442 6006 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:49:57.509032 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502446 6006 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:49:57.509032 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502450 6006 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:49:57.509032 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502453 6006 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:49:57.509032 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502456 6006 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:49:57.509032 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502459 6006 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:49:57.509032 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502462 6006 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:49:57.509032 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502465 6006 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:49:57.509032 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502468 6006 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:49:57.509032 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502473 6006 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:49:57.509032 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502477 6006 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:49:57.509643 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502480 6006 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:49:57.509643 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502484 6006 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:49:57.509643 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502487 6006 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:49:57.509643 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502491 6006 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:49:57.509643 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502494 6006 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:49:57.509643 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502498 6006 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:49:57.509643 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502501 6006 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:49:57.509643 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502506 6006 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:49:57.509643 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502510 6006 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:49:57.509643 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502513 6006 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:49:57.509643 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502516 6006 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:49:57.509643 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502519 6006 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:49:57.509643 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502523 6006 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:49:57.509643 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502527 6006 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:49:57.509643 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502530 6006 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:49:57.509643 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502533 6006 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:49:57.509643 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502536 6006 flags.go:64] FLAG: --enable-server="true" Oct 30 21:49:57.509643 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502539 6006 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:49:57.509643 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502544 6006 flags.go:64] FLAG: --event-burst="100" Oct 30 21:49:57.509643 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502548 6006 flags.go:64] FLAG: --event-qps="50" Oct 30 21:49:57.509643 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502551 6006 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:49:57.509643 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502554 6006 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:49:57.509643 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502557 6006 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:49:57.509643 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502569 6006 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:49:57.510306 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502573 6006 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:49:57.510306 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502576 6006 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:49:57.510306 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502580 6006 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:49:57.510306 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502583 6006 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:49:57.510306 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502586 6006 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:49:57.510306 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502589 6006 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:49:57.510306 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502592 6006 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:49:57.510306 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502597 6006 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:49:57.510306 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502600 6006 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:49:57.510306 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502603 6006 flags.go:64] FLAG: --feature-gates="" Oct 30 21:49:57.510306 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502607 6006 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:49:57.510306 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502610 6006 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:49:57.510306 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502614 6006 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:49:57.510306 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502617 6006 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:49:57.510306 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502620 6006 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:49:57.510306 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502625 6006 flags.go:64] FLAG: --help="false" Oct 30 21:49:57.510306 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502628 6006 flags.go:64] FLAG: --hostname-override="ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:49:57.510306 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502632 6006 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:49:57.510306 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502635 6006 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:49:57.510306 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502638 6006 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:49:57.510306 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502642 6006 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:49:57.510306 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502645 6006 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:49:57.510306 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502648 6006 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:49:57.510895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502651 6006 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:49:57.510895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502654 6006 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:49:57.510895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502657 6006 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:49:57.510895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502660 6006 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:49:57.510895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502663 6006 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:49:57.510895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502666 6006 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:49:57.510895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502669 6006 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:49:57.510895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502672 6006 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:49:57.510895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502676 6006 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:49:57.510895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502679 6006 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:49:57.510895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502682 6006 flags.go:64] FLAG: --lock-file="" Oct 30 21:49:57.510895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502685 6006 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:49:57.510895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502688 6006 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:49:57.510895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502691 6006 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:49:57.510895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502696 6006 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:49:57.510895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502699 6006 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:49:57.510895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502704 6006 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:49:57.510895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502707 6006 flags.go:64] FLAG: --logging-format="text" Oct 30 21:49:57.510895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502710 6006 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:49:57.510895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502714 6006 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:49:57.510895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502717 6006 flags.go:64] FLAG: --manifest-url="" Oct 30 21:49:57.510895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502720 6006 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:49:57.510895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502725 6006 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:49:57.510895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502728 6006 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:49:57.510895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502734 6006 flags.go:64] FLAG: --max-pods="110" Oct 30 21:49:57.511523 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502737 6006 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:49:57.511523 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502740 6006 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:49:57.511523 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502743 6006 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:49:57.511523 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502747 6006 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:49:57.511523 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502750 6006 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:49:57.511523 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502753 6006 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:49:57.511523 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502756 6006 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:49:57.511523 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502764 6006 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:49:57.511523 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502767 6006 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:49:57.511523 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502770 6006 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:49:57.511523 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502773 6006 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:49:57.511523 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502776 6006 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:49:57.511523 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502779 6006 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:49:57.511523 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502782 6006 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:49:57.511523 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502785 6006 flags.go:64] FLAG: --port="10250" Oct 30 21:49:57.511523 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502788 6006 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:49:57.511523 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502791 6006 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-06c6e410af1c4c253" Oct 30 21:49:57.511523 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502795 6006 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:49:57.511523 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502798 6006 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:49:57.511523 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502802 6006 flags.go:64] FLAG: --register-node="true" Oct 30 21:49:57.511523 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502805 6006 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:49:57.511523 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502809 6006 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:49:57.511523 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502812 6006 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:49:57.511523 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502816 6006 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:49:57.511523 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502819 6006 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:49:57.511523 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502822 6006 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:49:57.512185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502826 6006 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:49:57.512185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502829 6006 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:49:57.512185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502832 6006 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:49:57.512185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502835 6006 flags.go:64] FLAG: --runonce="false" Oct 30 21:49:57.512185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502838 6006 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:49:57.512185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502842 6006 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:49:57.512185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502846 6006 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:49:57.512185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502849 6006 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:49:57.512185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502852 6006 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:49:57.512185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502855 6006 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:49:57.512185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502858 6006 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:49:57.512185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502861 6006 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:49:57.512185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502864 6006 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:49:57.512185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502873 6006 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:49:57.512185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502876 6006 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:49:57.512185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502879 6006 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:49:57.512185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502882 6006 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:49:57.512185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502885 6006 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:49:57.512185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502888 6006 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:49:57.512185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502893 6006 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:49:57.512185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502896 6006 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:49:57.512185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502899 6006 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:49:57.512185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502906 6006 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:49:57.512185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502908 6006 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:49:57.512185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502917 6006 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:49:57.512814 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502920 6006 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:49:57.512814 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502923 6006 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:49:57.512814 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502926 6006 flags.go:64] FLAG: --v="2" Oct 30 21:49:57.512814 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502931 6006 flags.go:64] FLAG: --version="false" Oct 30 21:49:57.512814 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502936 6006 flags.go:64] FLAG: --vmodule="" Oct 30 21:49:57.512814 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502940 6006 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:49:57.512814 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.502944 6006 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:49:57.512814 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503109 6006 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:49:57.512814 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503113 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:49:57.512814 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503116 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:49:57.512814 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503119 6006 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:49:57.512814 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503121 6006 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:49:57.512814 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503125 6006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:49:57.512814 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503128 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:49:57.512814 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503131 6006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:49:57.512814 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503134 6006 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:49:57.512814 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503136 6006 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:49:57.512814 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503139 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:49:57.512814 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503142 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:49:57.512814 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503145 6006 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:49:57.512814 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503148 6006 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:49:57.513397 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503151 6006 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:49:57.513397 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503154 6006 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:49:57.513397 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503157 6006 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:49:57.513397 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503160 6006 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:49:57.513397 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503162 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:49:57.513397 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503165 6006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:49:57.513397 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503168 6006 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:49:57.513397 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503170 6006 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:49:57.513397 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503174 6006 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:49:57.513397 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503178 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:49:57.513397 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503181 6006 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:49:57.513397 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503184 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:49:57.513397 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503187 6006 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:49:57.513397 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503190 6006 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:49:57.513397 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503193 6006 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:49:57.513397 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503197 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:49:57.513397 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503200 6006 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:49:57.513397 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503203 6006 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:49:57.513397 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503206 6006 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:49:57.513875 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503208 6006 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:49:57.513875 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503211 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:49:57.513875 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503214 6006 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:49:57.513875 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503216 6006 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:49:57.513875 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503220 6006 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:49:57.513875 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503223 6006 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:49:57.513875 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503226 6006 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:49:57.513875 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503229 6006 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:49:57.513875 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503231 6006 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:49:57.513875 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503234 6006 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:49:57.513875 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503237 6006 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:49:57.513875 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503239 6006 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:49:57.513875 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503242 6006 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:49:57.513875 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503245 6006 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:49:57.513875 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503264 6006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:49:57.513875 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503267 6006 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:49:57.513875 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503270 6006 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:49:57.513875 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503273 6006 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:49:57.513875 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503275 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:49:57.513875 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503278 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:49:57.514452 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503281 6006 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:49:57.514452 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503284 6006 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:49:57.514452 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503286 6006 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:49:57.514452 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503289 6006 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:49:57.514452 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503291 6006 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:49:57.514452 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503295 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:49:57.514452 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503298 6006 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:49:57.514452 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503300 6006 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:49:57.514452 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503304 6006 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:49:57.514452 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503309 6006 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:49:57.514452 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503313 6006 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:49:57.514452 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503317 6006 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:49:57.514452 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503320 6006 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:49:57.514452 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503324 6006 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:49:57.514452 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503327 6006 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:49:57.514452 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503329 6006 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:49:57.514452 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503334 6006 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:49:57.514452 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503336 6006 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:49:57.514452 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503339 6006 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:49:57.515160 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503342 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:49:57.515160 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503344 6006 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:49:57.515160 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503347 6006 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:49:57.515160 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503350 6006 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:49:57.515160 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503352 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:49:57.515160 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503355 6006 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:49:57.515160 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503358 6006 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:49:57.515160 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503360 6006 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:49:57.515160 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503363 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:49:57.515160 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503365 6006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:49:57.515160 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503368 6006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:49:57.515160 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503371 6006 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:49:57.515160 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503373 6006 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:49:57.515160 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503376 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:49:57.515160 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503379 6006 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:49:57.515160 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503381 6006 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:49:57.515160 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503384 6006 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:49:57.515160 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503387 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:49:57.515160 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503389 6006 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:49:57.515160 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503393 6006 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:49:57.515890 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503395 6006 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:49:57.515890 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503399 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:49:57.515890 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503402 6006 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:49:57.515890 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503405 6006 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:49:57.515890 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503408 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:49:57.515890 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503411 6006 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:49:57.515890 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503413 6006 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:49:57.515890 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503416 6006 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:49:57.515890 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503419 6006 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:49:57.515890 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503422 6006 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:49:57.515890 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503425 6006 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:49:57.515890 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503428 6006 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:49:57.515890 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503430 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:49:57.515890 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503433 6006 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:49:57.515890 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.503459 6006 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:49:57.515890 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503502 6006 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:49:57.516445 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503507 6006 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:49:57.516445 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503511 6006 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:49:57.516445 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503514 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:49:57.516445 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503517 6006 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:49:57.516445 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503520 6006 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:49:57.516445 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503522 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:49:57.516445 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503525 6006 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:49:57.516445 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503528 6006 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:49:57.516445 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503530 6006 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:49:57.516445 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503533 6006 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:49:57.516445 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503535 6006 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:49:57.516445 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503538 6006 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:49:57.516445 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503542 6006 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:49:57.516445 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503545 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:49:57.516445 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503547 6006 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:49:57.516445 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503550 6006 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:49:57.516445 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503553 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:49:57.516445 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503566 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:49:57.516445 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503571 6006 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:49:57.516969 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503574 6006 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:49:57.516969 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503577 6006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:49:57.516969 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503579 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:49:57.516969 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503583 6006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:49:57.516969 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503585 6006 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:49:57.516969 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503588 6006 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:49:57.516969 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503592 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:49:57.516969 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503595 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:49:57.516969 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503598 6006 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:49:57.516969 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503600 6006 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:49:57.516969 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503604 6006 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:49:57.516969 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503606 6006 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:49:57.516969 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503609 6006 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:49:57.516969 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503612 6006 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:49:57.516969 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503615 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:49:57.516969 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503617 6006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:49:57.516969 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503620 6006 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:49:57.516969 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503623 6006 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:49:57.516969 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503627 6006 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:49:57.517508 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503631 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:49:57.517508 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503635 6006 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:49:57.517508 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503638 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:49:57.517508 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503641 6006 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:49:57.517508 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503644 6006 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:49:57.517508 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503648 6006 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:49:57.517508 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503651 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:49:57.517508 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503654 6006 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:49:57.517508 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503657 6006 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:49:57.517508 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503659 6006 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:49:57.517508 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503662 6006 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:49:57.517508 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503670 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:49:57.517508 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503675 6006 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:49:57.517508 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503678 6006 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:49:57.517508 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503681 6006 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:49:57.517508 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503683 6006 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:49:57.517508 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503686 6006 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:49:57.517508 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503689 6006 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:49:57.517508 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503692 6006 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:49:57.517508 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503696 6006 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:49:57.518170 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503699 6006 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:49:57.518170 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503701 6006 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:49:57.518170 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503704 6006 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:49:57.518170 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503707 6006 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:49:57.518170 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503709 6006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:49:57.518170 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503712 6006 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:49:57.518170 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503715 6006 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:49:57.518170 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503717 6006 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:49:57.518170 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503720 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:49:57.518170 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503722 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:49:57.518170 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503725 6006 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:49:57.518170 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503728 6006 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:49:57.518170 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503730 6006 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:49:57.518170 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503733 6006 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:49:57.518170 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503736 6006 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:49:57.518170 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503738 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:49:57.518170 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503741 6006 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:49:57.518170 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503744 6006 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:49:57.518170 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503746 6006 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:49:57.518170 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503751 6006 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:49:57.518771 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503755 6006 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:49:57.518771 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503758 6006 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:49:57.518771 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503761 6006 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:49:57.518771 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503764 6006 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:49:57.518771 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503773 6006 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:49:57.518771 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503776 6006 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:49:57.518771 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503779 6006 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:49:57.518771 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503781 6006 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:49:57.518771 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503784 6006 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:49:57.518771 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503787 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:49:57.518771 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503789 6006 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:49:57.518771 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503793 6006 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:49:57.518771 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503796 6006 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:49:57.518771 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503799 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:49:57.518771 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503801 6006 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:49:57.518771 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503804 6006 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:49:57.518771 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503807 6006 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:49:57.518771 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503810 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:49:57.518771 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503812 6006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:49:57.519434 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503815 6006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:49:57.519434 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503817 6006 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:49:57.519434 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503820 6006 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:49:57.519434 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503823 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:49:57.519434 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503826 6006 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:49:57.519434 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503828 6006 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:49:57.519434 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503831 6006 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:49:57.519434 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.503834 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:49:57.519434 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.509440 6006 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:49:57.519434 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.509457 6006 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:49:57.519434 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.509481 6006 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:49:57.519434 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.509486 6006 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:49:57.519434 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509542 6006 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:49:57.519434 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509547 6006 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:49:57.519434 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509550 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:49:57.519434 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509553 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:49:57.519434 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509556 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:49:57.519434 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509560 6006 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:49:57.519434 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509562 6006 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:49:57.519984 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509565 6006 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:49:57.519984 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509570 6006 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:49:57.519984 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509574 6006 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:49:57.519984 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509578 6006 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:49:57.519984 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509581 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:49:57.519984 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509585 6006 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:49:57.519984 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509589 6006 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:49:57.519984 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509592 6006 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:49:57.519984 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509595 6006 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:49:57.519984 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509603 6006 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:49:57.519984 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509606 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:49:57.519984 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509609 6006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:49:57.519984 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509612 6006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:49:57.519984 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509615 6006 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:49:57.519984 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509618 6006 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:49:57.519984 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509620 6006 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:49:57.519984 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509623 6006 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:49:57.519984 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509626 6006 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:49:57.519984 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509629 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:49:57.520579 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509632 6006 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:49:57.520579 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509634 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:49:57.520579 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509637 6006 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:49:57.520579 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509640 6006 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:49:57.520579 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509643 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:49:57.520579 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509646 6006 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:49:57.520579 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509649 6006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:49:57.520579 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509652 6006 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:49:57.520579 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509655 6006 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:49:57.520579 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509657 6006 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:49:57.520579 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509660 6006 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:49:57.520579 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509663 6006 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:49:57.520579 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509665 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:49:57.520579 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509668 6006 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:49:57.520579 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509671 6006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:49:57.520579 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509674 6006 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:49:57.520579 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509676 6006 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:49:57.520579 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509680 6006 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:49:57.520579 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509682 6006 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:49:57.520579 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509685 6006 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:49:57.521071 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509688 6006 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:49:57.521071 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509691 6006 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:49:57.521071 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509694 6006 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:49:57.521071 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509696 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:49:57.521071 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509699 6006 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:49:57.521071 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509702 6006 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:49:57.521071 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509705 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:49:57.521071 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509707 6006 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:49:57.521071 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509711 6006 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:49:57.521071 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509713 6006 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:49:57.521071 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509716 6006 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:49:57.521071 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509719 6006 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:49:57.521071 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509721 6006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:49:57.521071 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509724 6006 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:49:57.521071 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509727 6006 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:49:57.521071 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509729 6006 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:49:57.521071 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509732 6006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:49:57.521071 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509735 6006 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:49:57.521071 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509737 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:49:57.521071 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509740 6006 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:49:57.521628 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509742 6006 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:49:57.521628 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509745 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:49:57.521628 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509748 6006 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:49:57.521628 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509750 6006 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:49:57.521628 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509753 6006 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:49:57.521628 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509756 6006 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:49:57.521628 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509759 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:49:57.521628 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509762 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:49:57.521628 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509765 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:49:57.521628 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509767 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:49:57.521628 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509770 6006 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:49:57.521628 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509773 6006 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:49:57.521628 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509775 6006 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:49:57.521628 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509779 6006 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:49:57.521628 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509782 6006 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:49:57.521628 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509785 6006 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:49:57.521628 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509787 6006 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:49:57.521628 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509790 6006 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:49:57.521628 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509792 6006 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:49:57.521628 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509795 6006 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:49:57.522154 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509798 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:49:57.522154 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509800 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:49:57.522154 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509803 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:49:57.522154 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509806 6006 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:49:57.522154 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509808 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:49:57.522154 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509811 6006 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:49:57.522154 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509814 6006 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:49:57.522154 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509816 6006 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:49:57.522154 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509819 6006 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:49:57.522154 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509822 6006 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:49:57.522154 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509824 6006 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:49:57.522154 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509827 6006 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:49:57.522154 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509829 6006 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:49:57.522154 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509832 6006 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:49:57.522154 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509836 6006 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:49:57.522154 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509840 6006 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:49:57.522154 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509843 6006 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:49:57.522154 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509846 6006 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:49:57.522154 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509849 6006 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:49:57.522703 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.509852 6006 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:49:57.522703 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.509880 6006 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:49:57.522703 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510027 6006 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:49:57.522703 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510032 6006 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:49:57.522703 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510036 6006 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:49:57.522703 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510039 6006 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:49:57.522703 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510042 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:49:57.522703 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510045 6006 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:49:57.522703 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510048 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:49:57.522703 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510051 6006 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:49:57.522703 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510054 6006 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:49:57.522703 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510056 6006 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:49:57.522703 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510059 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:49:57.522703 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510062 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:49:57.522703 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510064 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:49:57.522703 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510067 6006 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:49:57.523147 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510069 6006 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:49:57.523147 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510072 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:49:57.523147 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510075 6006 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:49:57.523147 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510078 6006 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:49:57.523147 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510081 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:49:57.523147 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510084 6006 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:49:57.523147 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510088 6006 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:49:57.523147 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510092 6006 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:49:57.523147 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510095 6006 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:49:57.523147 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510098 6006 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:49:57.523147 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510100 6006 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:49:57.523147 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510103 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:49:57.523147 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510106 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:49:57.523147 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510108 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:49:57.523147 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510111 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:49:57.523147 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510114 6006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:49:57.523147 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510117 6006 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:49:57.523147 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510119 6006 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:49:57.523147 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510122 6006 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:49:57.523806 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510125 6006 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:49:57.523806 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510127 6006 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:49:57.523806 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510130 6006 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:49:57.523806 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510132 6006 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:49:57.523806 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510135 6006 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:49:57.523806 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510138 6006 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:49:57.523806 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510141 6006 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:49:57.523806 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510144 6006 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:49:57.523806 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510146 6006 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:49:57.523806 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510151 6006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:49:57.523806 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510155 6006 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:49:57.523806 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510159 6006 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:49:57.523806 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510163 6006 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:49:57.523806 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510166 6006 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:49:57.523806 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510169 6006 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:49:57.523806 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510172 6006 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:49:57.523806 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510174 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:49:57.523806 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510177 6006 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:49:57.523806 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510179 6006 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:49:57.524638 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510182 6006 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:49:57.524638 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510185 6006 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:49:57.524638 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510187 6006 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:49:57.524638 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510190 6006 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:49:57.524638 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510193 6006 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:49:57.524638 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510195 6006 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:49:57.524638 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510198 6006 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:49:57.524638 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510201 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:49:57.524638 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510204 6006 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:49:57.524638 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510206 6006 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:49:57.524638 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510209 6006 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:49:57.524638 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510212 6006 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:49:57.524638 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510215 6006 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:49:57.524638 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510218 6006 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:49:57.524638 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510220 6006 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:49:57.524638 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510223 6006 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:49:57.524638 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510225 6006 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:49:57.524638 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510228 6006 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:49:57.524638 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510231 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:49:57.524638 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510234 6006 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:49:57.525242 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510237 6006 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:49:57.525242 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510240 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:49:57.525242 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510244 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:49:57.525242 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510269 6006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:49:57.525242 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510272 6006 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:49:57.525242 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510275 6006 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:49:57.525242 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510278 6006 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:49:57.525242 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510281 6006 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:49:57.525242 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510284 6006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:49:57.525242 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510286 6006 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:49:57.525242 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510289 6006 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:49:57.525242 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510292 6006 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:49:57.525242 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510294 6006 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:49:57.525242 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510297 6006 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:49:57.525242 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510300 6006 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:49:57.525242 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510302 6006 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:49:57.525242 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510305 6006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:49:57.525242 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510308 6006 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:49:57.525242 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510311 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:49:57.525754 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510314 6006 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:49:57.525754 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510317 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:49:57.525754 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510320 6006 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:49:57.525754 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510323 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:49:57.525754 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510325 6006 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:49:57.525754 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510328 6006 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:49:57.525754 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510331 6006 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:49:57.525754 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510334 6006 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:49:57.525754 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510336 6006 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:49:57.525754 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510339 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:49:57.525754 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510342 6006 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:49:57.525754 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510345 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:49:57.525754 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510347 6006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:49:57.525754 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510351 6006 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:49:57.525754 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510354 6006 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:49:57.525754 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.510379 6006 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:49:57.526151 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510413 6006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:49:57.526151 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510419 6006 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:49:57.526151 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510422 6006 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:49:57.526151 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510425 6006 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:49:57.526151 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510428 6006 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:49:57.526151 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510431 6006 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:49:57.526151 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510433 6006 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:49:57.526151 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510436 6006 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:49:57.526151 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510439 6006 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:49:57.526151 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510441 6006 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:49:57.526151 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510444 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:49:57.526151 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510447 6006 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:49:57.526151 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510450 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:49:57.526151 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510452 6006 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:49:57.526151 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510455 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:49:57.526151 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510457 6006 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:49:57.526151 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510460 6006 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:49:57.526151 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510463 6006 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:49:57.526151 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510466 6006 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:49:57.526719 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510468 6006 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:49:57.526719 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510471 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:49:57.526719 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510474 6006 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:49:57.526719 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510476 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:49:57.526719 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510479 6006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:49:57.526719 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510482 6006 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:49:57.526719 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510484 6006 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:49:57.526719 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510487 6006 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:49:57.526719 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510489 6006 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:49:57.526719 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510492 6006 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:49:57.526719 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510495 6006 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:49:57.526719 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510498 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:49:57.526719 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510501 6006 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:49:57.526719 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510503 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:49:57.526719 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510508 6006 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:49:57.526719 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510511 6006 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:49:57.526719 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510513 6006 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:49:57.526719 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510516 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:49:57.526719 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510518 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:49:57.527199 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510522 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:49:57.527199 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510524 6006 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:49:57.527199 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510527 6006 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:49:57.527199 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510529 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:49:57.527199 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510532 6006 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:49:57.527199 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510535 6006 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:49:57.527199 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510538 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:49:57.527199 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510541 6006 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:49:57.527199 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510545 6006 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:49:57.527199 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510548 6006 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:49:57.527199 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510551 6006 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:49:57.527199 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510555 6006 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:49:57.527199 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510558 6006 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:49:57.527199 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510560 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:49:57.527199 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510563 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:49:57.527199 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510565 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:49:57.527199 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510568 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:49:57.527199 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510571 6006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:49:57.527199 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510573 6006 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:49:57.527726 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510576 6006 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:49:57.527726 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510578 6006 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:49:57.527726 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510581 6006 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:49:57.527726 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510583 6006 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:49:57.527726 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510586 6006 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:49:57.527726 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510588 6006 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:49:57.527726 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510592 6006 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:49:57.527726 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510595 6006 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:49:57.527726 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510599 6006 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:49:57.527726 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510602 6006 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:49:57.527726 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510605 6006 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:49:57.527726 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510607 6006 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:49:57.527726 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510611 6006 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:49:57.527726 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510615 6006 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:49:57.527726 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510618 6006 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:49:57.527726 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510621 6006 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:49:57.527726 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510624 6006 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:49:57.527726 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510627 6006 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:49:57.527726 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510629 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:49:57.528494 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510632 6006 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:49:57.528494 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510635 6006 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:49:57.528494 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510637 6006 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:49:57.528494 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510640 6006 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:49:57.528494 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510643 6006 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:49:57.528494 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510645 6006 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:49:57.528494 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510648 6006 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:49:57.528494 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510650 6006 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:49:57.528494 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510653 6006 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:49:57.528494 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510656 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:49:57.528494 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510658 6006 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:49:57.528494 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510661 6006 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:49:57.528494 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510664 6006 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:49:57.528494 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510666 6006 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:49:57.528494 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510669 6006 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:49:57.528494 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510671 6006 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:49:57.528494 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510674 6006 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:49:57.528494 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510677 6006 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:49:57.528494 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510679 6006 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:49:57.528494 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510682 6006 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:49:57.529155 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510686 6006 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:49:57.529155 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510690 6006 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:49:57.529155 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510693 6006 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:49:57.529155 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510695 6006 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:49:57.529155 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510698 6006 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:49:57.529155 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510702 6006 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:49:57.529155 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510704 6006 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:49:57.529155 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510707 6006 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:49:57.529155 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510710 6006 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:49:57.529155 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:49:57.510712 6006 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:49:57.529155 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.510773 6006 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:49:57.529155 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.511749 6006 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 21:49:57.529155 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.511816 6006 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 21:49:57.529155 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.512014 6006 server.go:1014] "Starting client certificate rotation" Oct 30 21:49:57.529155 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.512160 6006 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:49:57.529155 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.512217 6006 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:43:44 +0000 UTC" deadline="2028-05-14 19:34:29.617899838 +0000 UTC" Oct 30 21:49:57.529616 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.512263 6006 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13485h44m32.105638827s" Oct 30 21:49:57.529616 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.512545 6006 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:49:57.529616 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.513676 6006 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:49:57.529616 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.515339 6006 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:49:57.529616 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.517648 6006 log.go:25] "Validated CRI v1 image API" Oct 30 21:49:57.529616 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.518415 6006 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:49:57.529616 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.521109 6006 fs.go:135] Filesystem UUIDs: map[3578952b-c51b-4da5-bbc6-ad5ddc55ef30:/dev/nvme0n1p4 49414ccb-8903-4060-bec4-bc8797a77cc7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 21:49:57.529845 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.521142 6006 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/114ac894e52562dbb5f4c1eb70b3149ea1ecd5934c7693552db163ded70cd2c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/114ac894e52562dbb5f4c1eb70b3149ea1ecd5934c7693552db163ded70cd2c9/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/163fe39a07496c2c1b0c6d137e35ddfe934972beac5729c3c172d8ceeb97c2d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/163fe39a07496c2c1b0c6d137e35ddfe934972beac5729c3c172d8ceeb97c2d3/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/180790b45601d52b4ff7416e7ab9c05ee09a770df969fc05917948b9dde6f85d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/180790b45601d52b4ff7416e7ab9c05ee09a770df969fc05917948b9dde6f85d/userdata/shm major:0 minor:432 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25c043a82f378ee1f768d77bc37682bee28ed6f94b374c58924c70d296371cda/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25c043a82f378ee1f768d77bc37682bee28ed6f94b374c58924c70d296371cda/userdata/shm major:0 minor:523 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2942f9ed8024d9ebefc206e92519f5b963850505ad500f668cc5035fdf6e7edf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2942f9ed8024d9ebefc206e92519f5b963850505ad500f668cc5035fdf6e7edf/userdata/shm major:0 minor:453 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2aca39e96a0a5359c3c6f3557b7fab773e71a0f6396b23c99359d1f48424db5d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2aca39e96a0a5359c3c6f3557b7fab773e71a0f6396b23c99359d1f48424db5d/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c831fa197eb040ab59649744e4fa44b632eda251171ae2c0653ba688edf1edd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c831fa197eb040ab59649744e4fa44b632eda251171ae2c0653ba688edf1edd/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/372b376c17a9437a2aaed178dc1027b6d2f6bd6e7a3642656f226e4ab82e94b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/372b376c17a9437a2aaed178dc1027b6d2f6bd6e7a3642656f226e4ab82e94b3/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/373986040c020db99e3077377d47cc599f2eb297977682f180fd47a3c2e1c1f5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/373986040c020db99e3077377d47cc599f2eb297977682f180fd47a3c2e1c1f5/userdata/shm major:0 minor:337 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/379969793eebc3c4a8ee7b787852642f2d0bab68b5e4b5c5ca7277ebc1584eb7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/379969793eebc3c4a8ee7b787852642f2d0bab68b5e4b5c5ca7277ebc1584eb7/userdata/shm major:0 minor:470 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d8709f5a2d9ef7a07b4bbf4c0ca1718a68539b27a9a989e532683c0467b8119/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d8709f5a2d9ef7a07b4bbf4c0ca1718a68539b27a9a989e532683c0467b8119/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49c17a497980cd740b0d8a74998b831e8d742944c245c732c35c7d2fafb86827/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49c17a497980cd740b0d8a74998b831e8d742944c245c732c35c7d2fafb86827/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d9c6673201363e110fb9941032ebf8c83fc19e1fe99fe2539d7b22f07db02c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d9c6673201363e110fb9941032ebf8c83fc19e1fe99fe2539d7b22f07db02c6/userdata/shm major:0 minor:467 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f020130a1364da047db4e8b77083514ccee215eae989a164c46e413faef9d03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f020130a1364da047db4e8b77083514ccee215eae989a164c46e413faef9d03/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7140c65db18f03b4c55457261bcf97ff25e415df1bc664655ff2f0d85916a07e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7140c65db18f03b4c55457261bcf97ff25e415df1bc664655ff2f0d85916a07e/userdata/shm major:0 minor:466 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/72df4ed43b9eb340b736a6f8f63ed88621c05e05d2c0630828120e93b3bac8c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/72df4ed43b9eb340b736a6f8f63ed88621c05e05d2c0630828120e93b3bac8c2/userdata/shm major:0 minor:460 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e9c3b7469df82e4470afca5fe1a589a7407545d3785a7c0026ed27ecc696413/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e9c3b7469df82e4470afca5fe1a589a7407545d3785a7c0026ed27ecc696413/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/807ca5626e858e6a12ee34390615264e14111f00424b2e4b7942a71865766972/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/807ca5626e858e6a12ee34390615264e14111f00424b2e4b7942a71865766972/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/928ed5956456ef15f9e219522ca60366bbf675bbaca85d60cb4ed63fd76dbbfa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/928ed5956456ef15f9e219522ca60366bbf675bbaca85d60cb4ed63fd76dbbfa/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ea7a7f39436bd95eb87aaa921bf467e4ccb983ecc7b9cb726badd8d5ea2b22e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ea7a7f39436bd95eb87aaa921bf467e4ccb983ecc7b9cb726badd8d5ea2b22e/userdata/shm major:0 minor:341 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f8402d180547bcff85e4b4c8359d241ae0dce587db164dda9246bdbf3ef7732/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f8402d180547bcff85e4b4c8359d241ae0dce587db164dda9246bdbf3ef7732/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a3983aaf8de073607eb578a07952b50edababd4cf514ebe150ae01940d0fbe8c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a3983aaf8de073607eb578a07952b50edababd4cf514ebe150ae01940d0fbe8c/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8f41036b33b5ee78d7094694eb5bbd534b732f7cc74ed189e023f1179486594/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8f41036b33b5ee78d7094694eb5bbd534b732f7cc74ed189e023f1179486594/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab93116a47c2064a9a3779d9cae4a89403f0cc9d15d2431c9f8b04382b35c13f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab93116a47c2064a9a3779d9cae4a89403f0cc9d15d2431c9f8b04382b35c13f/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/add64044c3798234efbd7b2232ce3f42b48efeb0e148bdbad067ad8b7b6ed9d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/add64044c3798234efbd7b2232ce3f42b48efeb0e148bdbad067ad8b7b6ed9d3/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1aee542906e797088cb6a6b0121f8a722ad89edfd3fc5a8bb6b9fd211dbd171/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1aee542906e797088cb6a6b0121f8a722ad89edfd3fc5a8bb6b9fd211dbd171/userdata/shm major:0 minor:384 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b6a2dd3b149a596b09c061b9b51c78112abf4aef1a98c011da199fb570c87008/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b6a2dd3b149a596b09c061b9b51c78112abf4aef1a98c011da199fb570c87008/userdata/shm major:0 minor:357 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba241dfb9dc382019190ceb8b1d6fdeddcc1972e09a124e0be2b603a4e0d2cf5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba241dfb9dc382019190ceb8b1d6fdeddcc1972e09a124e0be2b603a4e0d2cf5/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bbd26b51457e1453cc51ec773785a48957a5de44d7e58aaf6e47e64808180460/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bbd26b51457e1453cc51ec773785a48957a5de44d7e58aaf6e47e64808180460/userdata/shm major:0 minor:386 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c2fab8c8f94425955fda1227f35b73dad2362caa4ff1924c6bc2d05c5fd4f59e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c2fab8c8f94425955fda1227f35b73dad2362caa4ff1924c6bc2d05c5fd4f59e/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c59a8eea64518cccb015decbdeff9327a62f4223b713bc4278344fd1041ce992/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c59a8eea64518cccb015decbdeff9327a62f4223b713bc4278344fd1041ce992/userdata/shm major:0 minor:339 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c65f9e423b79ea4d8c7c6522536eb48c1550c1bb189aad40fd355f8cd78429ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c65f9e423b79ea4d8c7c6522536eb48c1550c1bb189aad40fd355f8cd78429ef/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d6c5d5133bfcf2acea021dfa7d0287a4297dbe1a1f03d2bb38e27b73b5ac71ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d6c5d5133bfcf2acea021dfa7d0287a4297dbe1a1f03d2bb38e27b73b5ac71ef/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dc529468b79e6b85d51b5854b713775940759b7a512c5c0ebacbe15b472060b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dc529468b79e6b85d51b5854b713775940759b7a512c5c0ebacbe15b472060b1/userdata/shm major:0 minor:372 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3fff7abd1c2d54ab14f33d63025e973a308ca7bb52976feceb7d4eab6d7ba1c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3fff7abd1c2d54ab14f33d63025e973a308ca7bb52976feceb7d4eab6d7ba1c/userdata/shm major:0 minor:464 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eab77923780ae46182d22025f2db7c4e2214c1db453230aeee370578f48f89a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eab77923780ae46182d22025f2db7c4e2214c1db453230aeee370578f48f89a2/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f839a6af4cc59cc154537387101d77f74429e09b52a630505d1737c8a234134f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f839a6af4cc59cc154537387101d77f74429e09b52a630505d1737c8a234134f/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0dce5b38-e1a7-41e3-8100-96af4e3c2b06/volumes/kubernetes.io~projected/kube-api-access-jr7zd:{mountpoint:/var/lib/kubelet/pods/0dce5b38-e1a7-41e3-8100-96af4e3c2b06/volumes/kubernetes.io~projected/kube-api-access-jr7zd major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11109f61-f4f0-4253-aacb-886c353a5772/volumes/kubernetes.io~projected/kube-api-access-vgccp:{mountpoint:/var/lib/kubelet/pods/11109f61-f4f0-4253-aacb-886c353a5772/volumes/kubernetes.io~projected/kube-api-access-vgccp major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11109f61-f4f0-4253-aacb-886c353a5772/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/11109f61-f4f0-4253-aacb-886c353a5772/volumes/kubernetes.io~secret/serving-cert major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11ba3d2b-2d12-458c-9984-1dd82c275e10/volumes/kubernetes.io~projected/kube-api-access-58wrk:{mountpoint:/var/lib/kubelet/pods/11ba3d2b-2d12-458c-9984-1dd82c275e10/volumes/kubernetes.io~projected/kube-api-access-58wrk major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11ba3d2b-2d12-458c-9984-1dd82c275e10/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/11ba3d2b-2d12-458c-9984-1dd82c275e10/volumes/kubernetes.io~secret/default-certificate major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11ba3d2b-2d12-458c-9984-1dd82c275e10/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/11ba3d2b-2d12-458c-9984-1dd82c275e10/volumes/kubernetes.io~secret/metrics-certs major:0 minor:455 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11ba3d2b-2d12-458c-9984-1dd82c275e10/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/11ba3d2b-2d12-458c-9984-1dd82c275e10/volumes/kubernetes.io~secret/stats-auth major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13fd2c81-54ab-4afa-9afe-68094563bf08/volumes/kubernetes.io~projected/kube-api-access-vc5dn:{mountpoint:/var/lib/kubelet/pods/13fd2c81-54ab-4afa-9afe-68094563bf08/volumes/kubernetes.io~projected/kube-api-access-vc5dn major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4/volumes/kubernetes.io~projected/kube-api-access-7q7c2:{mountpoint:/var/lib/kubelet/pods/1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4/volumes/kubernetes.io~projected/kube-api-access-7q7c2 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d430f7e-00f1-4b9e-9427-3e9971cfff31/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/1d430f7e-00f1-4b9e-9427-3e9971cfff31/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/226c41d5-de84-4950-978f-a1f0b4cbc319/volumes/kubernetes.io~projected/kube-api-access-67x9l:{mountpoint:/var/lib/kubelet/pods/226c41d5-de84-4950-978f-a1f0b4cbc319/volumes/kubernetes.io~projected/kube-api-access-67x9l major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3458688b-8142-492c-ab8b-bffec754c8c8/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/3458688b-8142-492c-ab8b-bffec754c8c8/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:462 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3458688b-8142-492c-ab8b-bffec754c8c8/volumes/kubernetes.io~projected/kube-api-access-hxnj9:{mountpoint:/var/lib/kubelet/pods/3458688b-8142-492c-ab8b-bffec754c8c8/volumes/kubernetes.io~projected/kube-api-access-hxnj9 major:0 minor:463 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3458688b-8142-492c-ab8b-bffec754c8c8/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/3458688b-8142-492c-ab8b-bffec754c8c8/volumes/kubernetes.io~projected/registry-tls major:0 minor:457 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3458688b-8142-492c-ab8b-bffec754c8c8/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/3458688b-8142-492c-ab8b-bffec754c8c8/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:459 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3458688b-8142-492c-ab8b-bffec754c8c8/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/3458688b-8142-492c-ab8b-bffec754c8c8/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:458 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35a08208-7245-4e1b-9c0b-9e825a808365/volumes/kubernetes.io~projected/kube-api-access-h8258:{mountpoint:/var/lib/kubelet/pods/35a08208-7245-4e1b-9c0b-9e825a808365/volumes/kubernetes.io~projected/kube-api-access-h8258 major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35a08208-7245-4e1b-9c0b-9e825a808365/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/35a08208-7245-4e1b-9c0b-9e825a808365/volumes/kubernetes.io~secret/serving-cert major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3/volumes/kubernetes.io~projected/kube-api-access-hgd6c:{mountpoint:/var/lib/kubelet/pods/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3/volumes/kubernetes.io~projected/kube-api-access-hgd6c major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3/volumes/kubernetes.io~secret/metrics-tls major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4579f026-39bd-465c-b0db-41676370ba55/volumes/kubernetes.io~projected/kube-api-access-l8ctq:{mountpoint:/var/lib/kubelet/pods/4579f026-39bd-465c-b0db-41676370ba55/volumes/kubernetes.io~projected/kube-api-access-l8ctq major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47917159-29d7-45b9-bbf2-13cef4408de9/volumes/kubernetes.io~secret/networking-console-plugin-cert:{mountpoint:/var/lib/kubelet/pods/47917159-29d7-45b9-bbf2-13cef4408de9/volumes/kubernetes.io~secret/networking-console-plugin-cert major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/550e690a-605e-47a4-ba95-74bcd26e0cc3/volumes/kubernetes.io~projected/kube-api-access-kk7l4:{mountpoint:/var/lib/kubelet/pods/550e690a-605e-47a4-ba95-74bcd26e0cc3/volumes/kubernetes.io~projected/kube-api-access-kk7l4 major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b2dbc17-c725-4893-896e-4dd2ec2dbc13/volumes/kubernetes.io~projected/kube-api-access-826wq:{mountpoint:/var/lib/kubelet/pods/5b2dbc17-c725-4893-896e-4dd2ec2dbc13/volumes/kubernetes.io~projected/kube-api-access-826wq major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b2dbc17-c725-4893-896e-4dd2ec2dbc13/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/5b2dbc17-c725-4893-896e-4dd2ec2dbc13/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:427 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/volumes/kubernetes.io~projected/kube-api-access-lkjjr:{mountpoint:/var/lib/kubelet/pods/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/volumes/kubernetes.io~projected/kube-api-access-lkjjr major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64086350-55f6-4da5-a871-3f23c1755371/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/64086350-55f6-4da5-a871-3f23c1755371/volumes/kubernetes.io~secret/agent-certs major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e73e56e-9619-4b97-bf20-a6291acf6355/volumes/kubernetes.io~projected/kube-api-access-f54bb:{mountpoint:/var/lib/kubelet/pods/6e73e56e-9619-4b97-bf20-a6291acf6355/volumes/kubernetes.io~projected/kube-api-access-f54bb major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7aa1b80f-115e-4708-bc98-88d13ad66860/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/7aa1b80f-115e-4708-bc98-88d13ad66860/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7aa1b80f-115e-4708-bc98-88d13ad66860/volumes/kubernetes.io~projected/kube-api-access-qvxrm:{mountpoint:/var/lib/kubelet/pods/7aa1b80f-115e-4708-bc98-88d13ad66860/volumes/kubernetes.io~projected/kube-api-access-qvxrm major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7aa1b80f-115e-4708-bc98-88d13ad66860/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/7aa1b80f-115e-4708-bc98-88d13ad66860/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84bfdb37-6346-4384-8d97-d43aba6f03fe/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/84bfdb37-6346-4384-8d97-d43aba6f03fe/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84bfdb37-6346-4384-8d97-d43aba6f03fe/volumes/kubernetes.io~projected/kube-api-access-vss2v:{mountpoint:/var/lib/kubelet/pods/84bfdb37-6346-4384-8d97-d43aba6f03fe/volumes/kubernetes.io~projected/kube-api-access-vss2v major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84bfdb37-6346-4384-8d97-d43aba6f03fe/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/84bfdb37-6346-4384-8d97-d43aba6f03fe/volumes/kubernetes.io~projected/registry-tls major:0 minor:454 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84bfdb37-6346-4384-8d97-d43aba6f03fe/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/84bfdb37-6346-4384-8d97-d43aba6f03fe/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84bfdb37-6346-4384-8d97-d43aba6f03fe/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/84bfdb37-6346-4384-8d97-d43aba6f03fe/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b60900b-7a0e-4039-97af-c8157ae19ca0/volumes/kubernetes.io~projected/kube-api-access-fcm8z:{mountpoint:/var/lib/kubelet/pods/8b60900b-7a0e-4039-97af-c8157ae19ca0/volumes/kubernetes.io~projected/kube-api-access-fcm8z major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bf0fd8f-2294-4729-9f49-a151de8627aa/volumes/kubernetes.io~projected/kube-api-access-r2b29:{mountpoint:/var/lib/kubelet/pods/8bf0fd8f-2294-4729-9f49-a151de8627aa/volumes/kubernetes.io~projected/kube-api-access-r2b29 major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8df7e7fd-be92-4d37-af33-930b7b36cbe6/volumes/kubernetes.io~secret/tls-certificates:{mountpoint:/var/lib/kubelet/pods/8df7e7fd-be92-4d37-af33-930b7b36cbe6/volumes/kubernetes.io~secret/tls-certificates major:0 minor:519 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9161cdbd-c44f-4a8c-b918-107fc7001f3a/volumes/kubernetes.io~projected/kube-api-access-9hz67:{mountpoint:/var/lib/kubelet/pods/9161cdbd-c44f-4a8c-b918-107fc7001f3a/volumes/kubernetes.io~projected/kube-api-access-9hz67 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95097323-8907-45bc-8967-0de718adefbd/volumes/kubernetes.io~projected/kube-api-access-j58ln:{mountpoint:/var/lib/kubelet/pods/95097323-8907-45bc-8967-0de718adefbd/volumes/kubernetes.io~projected/kube-api-access-j58ln major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95097323-8907-45bc-8967-0de718adefbd/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls:{mountpoint:/var/lib/kubelet/pods/95097323-8907-45bc-8967-0de718adefbd/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls major:0 minor:449 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9df43c88-a01f-4ec7-a385-d5afcf6f9e6c/volumes/kubernetes.io~projected/kube-api-access-thhdq:{mountpoint:/var/lib/kubelet/pods/9df43c88-a01f-4ec7-a385-d5afcf6f9e6c/volumes/kubernetes.io~projected/kube-api-access-thhdq major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a32b60f2-3000-4fe8-a928-955de8c76c82/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/a32b60f2-3000-4fe8-a928-955de8c76c82/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a32b60f2-3000-4fe8-a928-955de8c76c82/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/a32b60f2-3000-4fe8-a928-955de8c76c82/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a32b60f2-3000-4fe8-a928-955de8c76c82/volumes/kubernetes.io~projected/kube-api-access-r6k4l:{mountpoint:/var/lib/kubelet/pods/a32b60f2-3000-4fe8-a928-955de8c76c82/volumes/kubernetes.io~projected/kube-api-access-r6k4l major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2f2c486-d661-4d59-981c-0b95a1f66fb4/volumes/kubernetes.io~projected/kube-api-access-sw4xv:{mountpoint:/var/lib/kubelet/pods/b2f2c486-d661-4d59-981c-0b95a1f66fb4/volumes/kubernetes.io~projected/kube-api-access-sw4xv major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd3b421a-9313-4f15-b81f-900717685fd6/volumes/kubernetes.io~projected/kube-api-access-j2qx6:{mountpoint:/var/lib/kubelet/pods/bd3b421a-9313-4f15-b81f-900717685fd6/volumes/kubernetes.io~projected/kube-api-access-j2qx6 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd3b421a-9313-4f15-b81f-900717685fd6/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/bd3b421a-9313-4f15-b81f-900717685fd6/volumes/kubernetes.io~secret/metrics-certs major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0cece9c-2008-49ce-87de-4804d4e7de08/volumes/kubernetes.io~projected/kube-api-access-j92gp:{mountpoint:/var/lib/kubelet/pods/c0cece9c-2008-49ce-87de-4804d4e7de08/volumes/kubernetes.io~projected/kube-api-access-j92gp major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0cece9c-2008-49ce-87de-4804d4e7de08/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c0cece9c-2008-49ce-87de-4804d4e7de08/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7fe5efb-7029-41b8-8bbe-f37f49410f27/volumes/kubernetes.io~projected/kube-api-access-fn7gh:{mountpoint:/var/lib/kubelet/pods/c7fe5efb-7029-41b8-8bbe-f37f49410f27/volumes/kubernetes.io~projected/kube-api-access-fn7gh major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7fe5efb-7029-41b8-8bbe-f37f49410f27/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/c7fe5efb-7029-41b8-8bbe-f37f49410f27/volumes/kubernetes.io~secret/cert major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce930258-60d9-4988-913c-2f3cb78eac8d/volumes/kubernetes.io~projected/kube-api-access-v5gt6:{mountpoint:/var/lib/kubelet/pods/ce930258-60d9-4988-913c-2f3cb78eac8d/volumes/kubernetes.io~projected/kube-api-access-v5gt6 major:0 minor:456 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ceedf44e-88c1-49cf-901e-75558703c361/volumes/kubernetes.io~projected/kube-api-access-s8pl2:{mountpoint:/var/lib/kubelet/pods/ceedf44e-88c1-49cf-901e-75558703c361/volumes/kubernetes.io~projected/kube-api-access-s8pl2 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2/volumes/kubernetes.io~projected/kube-api-access-9hnzg:{mountpoint:/var/lib/kubelet/pods/d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2/volumes/kubernetes.io~projected/kube-api-access-9hnzg major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2/volumes/kubernetes.io~secret/signing-key major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f71e935d-f091-4f16-820b-8d7fc632f003/volumes/kubernetes.io~projected/kube-api-access-vdhdv:{mountpoint:/var/lib/kubelet/pods/f71e935d-f091-4f16-820b-8d7fc632f003/volumes/kubernetes.io~projected/kube-api-access-vdhdv major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc3ff956-6466-46e3-b734-13a3ac148d36/volumes/kubernetes.io~projected/kube-api-access-k9lcg:{mountpoint:/var/lib/kubelet/pods/fc3ff956-6466-46e3-b734-13a3ac148d36/volumes/kubernetes.io~projected/kube-api-access-k9lcg major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc3ff956-6466-46e3-b734-13a3ac148d36/volumes/kubernetes.io~secret/samples-operator-tls:{mountpoint:/var/lib/kubelet/pods/fc3ff956-6466-46e3-b734-13a3ac148d36/volumes/kubernetes.io~secret/samples-operator-tls major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffa64e38-3f1b-49af-913f-139addefa359/volumes/kubernetes.io~projected/kube-api-access-5fgkx:{mountpoint:/var/lib/kubelet/pods/ffa64e38-3f1b-49af-913f-139addefa359/volumes/kubernetes.io~projected/kube-api-access-5fgkx major:0 minor:62 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/1359435086b3e44184d393face49ede3a91d82911b80fef222e98f0ac7f3b7f6/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/78b4bedc8aa086501582c69efef39a2938a652c45dabee9e0af18fc184a4e1ee/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/d7b0de4dd2720584f0d03d7d17e74e1da9c03f27c8988f23d6f96c5f17ef5c6d/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/f9377056a998b14fd61a298bc39d8a3a13262697e4c5436f3a720c2cfeaef6b1/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/f4ab63447dc85cc259003acf6dfe0bd407b98352ef4611832e0f1f94910e84d6/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/6b397f110d43a0cf108a8a326a63036169c51980c991853e69becd57a5ef995a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/b685403c3ab9b5b8f4edb6c350f3848e99520f68f9f8db81ba99dc984768a8d8/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/c4b6ba9cddac42eec0d2310a2225d674896e57b241e78ec2121ffd44b8ae7555/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/467353015c4459206a8e7223e8161aea3e8456187dd1800799c94723872ffdc3/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/3e17b07cabd4df5132070eb2e33ad88f3730e2d9bd1a88797c347a3b30b6046a/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/26f9f79116f2531ba78b26d2c85725eb3b46915c1e05ebd75cd399a2500fb00a/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/f797477a8e79433922bce47c54c2b4d994a0a0e544bd003124d066d4273333dd/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/4e2dac40a92e9bb3d106aeebb166c49a264f7f0c4e12db1543d87f6cc0a62ade/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/8bb6cdcb3a017efec705473450d1bd5ed6828fc7dac9a93d19b6312186cf8ff9/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/8b39d0b3ce6205cadf8d23055d228e6e126cf9de2a6b2d4660726360fb66500f/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/8e2f66180bc4eb9901cf45f5bbc87d9cf9cec79998a52d494c5bcbced11c8189/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/578cc79464474992275ac2b9c601434a7c072d97f07ae5d5fad2b83efbd403b4/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/c93e068d7232431dbc98cc1549840db01408fbe0fc7d360f20f137bd9fc24193/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/0092a2dbaab7051991c67e099512fc87379327d88439b38dc04c4b0417458094/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/b2e88cff05c923f68f227508c0e44d6ea59a08342e715a717f945bfc2cffc3e9/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/838c2f7c1da8e49a9edd6f15e7339d052eba540be423ddeb219df6e65bdf11f1/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/2d4fa8946e055acc6318b9c71ddd2cf17074bff6594564f85162748989de6471/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/13b7e23141aae255d0afe2b42c47df69cb221a774064070addb19376fcf61845/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/a5286bc7c757e34fcb4be8ea0a6d4a2e6f7800923478623d39113491c9105710/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/937fd6dd50258d2d7cd478f59f9fdb874f5655e3edf5bb7c63a2887f1acb4cbd/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/0893504ac28e8cf4475a88d0ae6b13936db87fda76dfdf6115795e5447d5e15b/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/7e6988ae8338df617a0a9e39ef7cfe28a9cbe7cd9506f5f19ed0e80854178a59/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/350e282aa4756a2f7cb4d9d07760b7ae45c1ff836393c85481be35f8b6e97ee9/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/e33568f68722bc33704b93698978939e682aec0a6e8d4922db9138e70eebbcb6/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/7235e5ede5e94e15e75efd89c977715e044f33b80b231baa88b8ce003d64632d/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/54490b3a27e8ec2ede10b89eecadb72412ff38203639c35902c57ce2d8dbe221/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/d34054c727d0e36357f029e2edd1e0a8bbc9295d263ac972fe818fd73b29e56a/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/df781fed1fdfab4522cff8cd6959864c08684bacaf0173129d8f70d20d823cd5/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/a369685037117e2409f98587699254d3b08ed30c1a205386f281d956684713fc/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/2ece648f8067f5a636a6944582f7f5020bdb23d68d36a48f9fef6183462f9962/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/59bf3aff3c02d7b732d97c81bc2cacfd66737988f26455078b1d1fd302eb55a0/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/a415f2fe2b02723afc589b9e6a05e0f9d4b9303db72dbba68a066a64932258f2/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/c99f37ad18a2c68d0ef45139b4fd2ef6128e73d2b3d46a075f7e630566c7a032/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/0cb04691f2fd19c1e8dfc0dc962fc06f4bbd558f8b2eeaf0613e167b4b95a1d1/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/026e9ef26a24b5f97ff03b0f7ae7ae4ba575c6bf95d2642310502c6201783bab/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/6d1af45c976a0b82dd91828c03ae262efa4a6db6bdd2a16176f9ea2b4cee89a8/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/6c5189f6e77ee8d7f4d98abdb2dfd96335632377992e715da2cc231c02f53421/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/d845a734fa4da4eb01c032cd44a75f187850501c528b900abcf64b9b1f228910/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/8e4158a63216147dca79eb36514014f38cff2ff31d3c6a40b646aafae826d7ef/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/df22d4c6f1d70d8ac983bf13f42fe37e9c16a3470a7befe0cb286cebcd29070d/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/5557be2c68277719c6288ec464e21224da8a5acde2e1b59fc37b03b7e29c1bd1/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/ec082115cc4b5868bbec82782be8867f18593c57b75954da71245d95c766b0c3/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/f5f279cca60996d131f996b202d8f372493313091dfc2620adc97cf74967f605/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/7c51bd3a90bd3590a8548bb1a354c929bb73bdf1d287d1da118d299466b202a3/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/c3a1d5526feb7c9a419d20698a7bc6668fa174f0f3306b88a14ff8e6a7e96444/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/86dff050aa7f6df4a8d877548fa1164cbd19e7848dafc2f75536227baa33c4c6/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/505988f1d545406d99826958e81e2363c43823e67678e36b3e953fcc2ea4c816/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/43d4c6ebf69eda10e8b3246e6d29bcf4e83ab3f57a1e640ade36acc5340f9601/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/db2eded63fe71f136d03011c770b179bd323ef39bb7bf49b2463c04d8d93410d/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/81d50c3bddc671391b7d8caf24bb481398c068e1de42af219cc4f9a6ad4a1647/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/214477c9dd0fee733a706aafdbe77fcd7cced561ae88ed2f7d5ad04dafd07325/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/e23365d1db9a2f5eebe07e9c969e654103ba8c5c383199b44291eb0066831fc2/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-417:{mountpoint:/var/lib/containers/storage/overlay/535c4421d8594c854392a028b263458c0fae2da86e925d57bf16c38c926f0b7d/merged major:0 minor:417 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/8ada8024821349189f4e82857f7b0502b0a2104b27e2d8b43a5362f0af8a1ef5/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-434:{mountpoint:/var/lib/containers/storage/overlay/66d11d5dddcddf44423bda6defc11f9b491abf4ad40d8d5715566fa0f370f0dd/merged major:0 minor:434 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/340572b8de6da049499a0f76ace655e0d5700465d2cfe0be66ea95ff0c08cdac/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-438:{mountpoint:/var/lib/containers/storage/overlay/af11c9451ab2205f6a9a7e0111fc31f11a20713ac056fdd25a07c7ebd795c86b/merged major:0 minor:438 fsType:overlay blockSize:0} overlay_0-444:{mountpoint:/var/lib/containers/storage/overlay/4310373e085eeb9118dc5cb660ef259b64beef9042d81515d176a54f8f48d353/merged major:0 minor:444 fsType:overlay blockSize:0} overlay_0-472:{mountpoint:/var/lib/containers/storage/overlay/31a504cbb1189b58bdf6029ddbde1ecb051b3f7a834d46d7a7d419cd83a43e56/merged major:0 minor:472 fsType:overlay blockSize:0} overlay_0-474:{mountpoint:/var/lib/containers/storage/overlay/cba9aa0a6b5f48ed83b76726b09beff9aa314b4b60abc79f11b754719761a944/merged major:0 minor:474 fsType:overlay blockSize:0} overlay_0-476:{mountpoint:/var/lib/containers/storage/overlay/0d4cc34d97e7a0889fe7e71a0a0e53892d134d652a5acb5ae472fe5e2bcf6052/merged major:0 minor:476 fsType:overlay blockSize:0} overlay_0-478:{mountpoint:/var/lib/containers/storage/overlay/c922cf68506a1dd372d46e58240bd21786f02d2d0156c126d27a9cf446f92fd1/merged major:0 minor:478 fsType:overlay blockSize:0} overlay_0-480:{mountpoint:/var/lib/containers/storage/overlay/99c5f75ff9301eee83d3758695b86abc43e11f0a9b9ca9a9958906fde7b38017/merged major:0 minor:480 fsType:overlay blockSize:0} overlay_0-482:{mountpoint:/var/lib/containers/storage/overlay/82af3ae508a51c84209612ccd18efe4b6d7d803e6d8f91115f5a9a6057a26ee8/merged major:0 minor:482 fsType:overlay blockSize:0} overlay_0-484:{mountpoint:/var/lib/containers/storage/overlay/bfe978693cced46ac4fac4dd164de2c99ba59f53c29c861720f8c75a2d79d0c3/merged major:0 minor:484 fsType:overlay blockSize:0} overlay_0-487:{mountpoint:/var/lib/containers/storage/overlay/d0e657f494a16dd359ed795016644277dc6fdcae1ea64fb8d3cd482396b9ae3e/merged major:0 minor:487 fsType:overlay blockSize:0} overlay_0-492:{mountpoint:/var/lib/containers/storage/overlay/9a6076663c6bd5dbf45d13a658b9f19c845c0b8de46571446d986b61f24545b6/merged major:0 minor:492 fsType:overlay blockSize:0} overlay_0-503:{mountpoint:/var/lib/containers/storage/overlay/23c9727b2f790a15ca4d9dc53b6545857c47fda0471f4c1e49ff4f3ad1fa7953/merged major:0 minor:503 fsType:overlay blockSize:0} overlay_0-505:{mountpoint:/var/lib/containers/storage/overlay/cbb126bcd8ddb24efb98e2702cd4e6b309e7cebca5db5b9c7c5ce3817670412c/merged major:0 minor:505 fsType:overlay blockSize:0} overlay_0-507:{mountpoint:/var/lib/containers/storage/overlay/9657c3a836e8a6e6db657d2f2411ba30b195cc2ad491874be1f4297ad9e74231/merged major:0 minor:507 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/a192eaac54d2c7b6aa1000835b8c6d9ebdf112807ad7c4cee64961f64e262481/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-513:{mountpoint:/var/lib/containers/storage/overlay/4adcffa4bb25a22f6e04d5bd7a3bf605ad5583439a327ccc69b97a46a50c553c/merged major:0 minor:513 fsType:overlay blockSize:0} overlay_0-525:{mountpoint:/var/lib/containers/storage/overlay/648a4c939b6146d29855de9699379ae45348db9175d82430586ce23b58b6ac8b/merged major:0 minor:525 fsType:overlay blockSize:0} overlay_0-527:{mountpoint:/var/lib/containers/storage/overlay/ed7efca750821b6efb4fce94530455e495d62b27067e7a0eb469321eb842a571/merged major:0 minor:527 fsType:overlay blockSize:0} overlay_0-529:{mountpoint:/var/lib/containers/storage/overlay/63168d53313d7a656cc69f0982c4f76bd917db5d78f7ba130d87ea323ba4585c/merged major:0 minor:529 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/6435194dff5d358dbd0f22c37648083c9473577c1696e1cda291907b9d457033/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/36386d44b457fb51aaa26d56e64e7f4e9a1e2f31308aed6c4602f08902da4a7d/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/ff2b1df437b7745626a5e0e5d2a662cf1db3f675949f4753ebffaa3c4e73050b/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/994082ef7a3da5f7a7ab4a851932622f7527f4f7ed416304d37d2c167f24388a/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/56f9e717f46f0d3641f20de98c94f106ad6554cc5e2c0fca82b8a6691ab3e015/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/91baea7b3ab9ddefc55d43d3006d2de4f338c0a70033f0c2adb2956d9db7fd17/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/c8f39ad45fed905967ac0327d22a01661c9ee524e9035bca456483ffea8e3989/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/132769d1b629308e80ae0c0d5a7574d79a5a44656de6def1d4d98508cae12645/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/846c59e9f07f15812d8f2fd40cbd19866a68fb2b9134b6203a551dbc94a30265/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 21:49:57.533722 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.533240 6006 manager.go:217] Machine: {Timestamp:2026-10-30 21:49:57.532755942 +0000 UTC m=+0.084932234 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3088065 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2456eceec74055364afe40911ad25f SystemUUID:ec2456ec-eec7-4055-364a-fe40911ad25f BootID:12542b92-67c7-4e79-bcd5-56086dc469da Filesystems:[{Device:/var/lib/kubelet/pods/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-417 DeviceMajor:0 DeviceMinor:417 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b2dbc17-c725-4893-896e-4dd2ec2dbc13/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:427 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-487 DeviceMajor:0 DeviceMinor:487 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7aa1b80f-115e-4708-bc98-88d13ad66860/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-482 DeviceMajor:0 DeviceMinor:482 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-529 DeviceMajor:0 DeviceMinor:529 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f71e935d-f091-4f16-820b-8d7fc632f003/volumes/kubernetes.io~projected/kube-api-access-vdhdv DeviceMajor:0 DeviceMinor:335 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b6a2dd3b149a596b09c061b9b51c78112abf4aef1a98c011da199fb570c87008/userdata/shm DeviceMajor:0 DeviceMinor:357 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8df7e7fd-be92-4d37-af33-930b7b36cbe6/volumes/kubernetes.io~secret/tls-certificates DeviceMajor:0 DeviceMinor:519 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84bfdb37-6346-4384-8d97-d43aba6f03fe/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:202 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a3983aaf8de073607eb578a07952b50edababd4cf514ebe150ae01940d0fbe8c/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d8709f5a2d9ef7a07b4bbf4c0ca1718a68539b27a9a989e532683c0467b8119/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3458688b-8142-492c-ab8b-bffec754c8c8/volumes/kubernetes.io~projected/kube-api-access-hxnj9 DeviceMajor:0 DeviceMinor:463 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c7fe5efb-7029-41b8-8bbe-f37f49410f27/volumes/kubernetes.io~projected/kube-api-access-fn7gh DeviceMajor:0 DeviceMinor:241 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/373986040c020db99e3077377d47cc599f2eb297977682f180fd47a3c2e1c1f5/userdata/shm DeviceMajor:0 DeviceMinor:337 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3fff7abd1c2d54ab14f33d63025e973a308ca7bb52976feceb7d4eab6d7ba1c/userdata/shm DeviceMajor:0 DeviceMinor:464 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-474 DeviceMajor:0 DeviceMinor:474 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/11ba3d2b-2d12-458c-9984-1dd82c275e10/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:210 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fc3ff956-6466-46e3-b734-13a3ac148d36/volumes/kubernetes.io~projected/kube-api-access-k9lcg DeviceMajor:0 DeviceMinor:224 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eab77923780ae46182d22025f2db7c4e2214c1db453230aeee370578f48f89a2/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-434 DeviceMajor:0 DeviceMinor:434 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3458688b-8142-492c-ab8b-bffec754c8c8/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:457 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/379969793eebc3c4a8ee7b787852642f2d0bab68b5e4b5c5ca7277ebc1584eb7/userdata/shm DeviceMajor:0 DeviceMinor:470 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4/volumes/kubernetes.io~projected/kube-api-access-7q7c2 DeviceMajor:0 DeviceMinor:64 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3/volumes/kubernetes.io~projected/kube-api-access-hgd6c DeviceMajor:0 DeviceMinor:230 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c2fab8c8f94425955fda1227f35b73dad2362caa4ff1924c6bc2d05c5fd4f59e/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-444 DeviceMajor:0 DeviceMinor:444 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11ba3d2b-2d12-458c-9984-1dd82c275e10/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:455 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3458688b-8142-492c-ab8b-bffec754c8c8/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:459 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e73e56e-9619-4b97-bf20-a6291acf6355/volumes/kubernetes.io~projected/kube-api-access-f54bb DeviceMajor:0 DeviceMinor:330 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-478 DeviceMajor:0 DeviceMinor:478 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-505 DeviceMajor:0 DeviceMinor:505 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-507 DeviceMajor:0 DeviceMinor:507 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9161cdbd-c44f-4a8c-b918-107fc7001f3a/volumes/kubernetes.io~projected/kube-api-access-9hz67 DeviceMajor:0 DeviceMinor:66 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba241dfb9dc382019190ceb8b1d6fdeddcc1972e09a124e0be2b603a4e0d2cf5/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9df43c88-a01f-4ec7-a385-d5afcf6f9e6c/volumes/kubernetes.io~projected/kube-api-access-thhdq DeviceMajor:0 DeviceMinor:197 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f8402d180547bcff85e4b4c8359d241ae0dce587db164dda9246bdbf3ef7732/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8b60900b-7a0e-4039-97af-c8157ae19ca0/volumes/kubernetes.io~projected/kube-api-access-fcm8z DeviceMajor:0 DeviceMinor:336 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fc3ff956-6466-46e3-b734-13a3ac148d36/volumes/kubernetes.io~secret/samples-operator-tls DeviceMajor:0 DeviceMinor:382 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84bfdb37-6346-4384-8d97-d43aba6f03fe/volumes/kubernetes.io~projected/kube-api-access-vss2v DeviceMajor:0 DeviceMinor:218 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:383 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3458688b-8142-492c-ab8b-bffec754c8c8/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:462 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-476 DeviceMajor:0 DeviceMinor:476 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2942f9ed8024d9ebefc206e92519f5b963850505ad500f668cc5035fdf6e7edf/userdata/shm DeviceMajor:0 DeviceMinor:453 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0dce5b38-e1a7-41e3-8100-96af4e3c2b06/volumes/kubernetes.io~projected/kube-api-access-jr7zd DeviceMajor:0 DeviceMinor:201 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11109f61-f4f0-4253-aacb-886c353a5772/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:225 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/180790b45601d52b4ff7416e7ab9c05ee09a770df969fc05917948b9dde6f85d/userdata/shm DeviceMajor:0 DeviceMinor:432 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1d430f7e-00f1-4b9e-9427-3e9971cfff31/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e9c3b7469df82e4470afca5fe1a589a7407545d3785a7c0026ed27ecc696413/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95097323-8907-45bc-8967-0de718adefbd/volumes/kubernetes.io~projected/kube-api-access-j58ln DeviceMajor:0 DeviceMinor:219 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c7fe5efb-7029-41b8-8bbe-f37f49410f27/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:235 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-472 DeviceMajor:0 DeviceMinor:472 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-503 DeviceMajor:0 DeviceMinor:503 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8f41036b33b5ee78d7094694eb5bbd534b732f7cc74ed189e023f1179486594/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/volumes/kubernetes.io~projected/kube-api-access-lkjjr DeviceMajor:0 DeviceMinor:222 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/114ac894e52562dbb5f4c1eb70b3149ea1ecd5934c7693552db163ded70cd2c9/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/47917159-29d7-45b9-bbf2-13cef4408de9/volumes/kubernetes.io~secret/networking-console-plugin-cert DeviceMajor:0 DeviceMinor:371 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bd3b421a-9313-4f15-b81f-900717685fd6/volumes/kubernetes.io~projected/kube-api-access-j2qx6 DeviceMajor:0 DeviceMinor:55 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab93116a47c2064a9a3779d9cae4a89403f0cc9d15d2431c9f8b04382b35c13f/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1aee542906e797088cb6a6b0121f8a722ad89edfd3fc5a8bb6b9fd211dbd171/userdata/shm DeviceMajor:0 DeviceMinor:384 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/72df4ed43b9eb340b736a6f8f63ed88621c05e05d2c0630828120e93b3bac8c2/userdata/shm DeviceMajor:0 DeviceMinor:460 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d6c5d5133bfcf2acea021dfa7d0287a4297dbe1a1f03d2bb38e27b73b5ac71ef/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ceedf44e-88c1-49cf-901e-75558703c361/volumes/kubernetes.io~projected/kube-api-access-s8pl2 DeviceMajor:0 DeviceMinor:61 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/add64044c3798234efbd7b2232ce3f42b48efeb0e148bdbad067ad8b7b6ed9d3/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4579f026-39bd-465c-b0db-41676370ba55/volumes/kubernetes.io~projected/kube-api-access-l8ctq DeviceMajor:0 DeviceMinor:231 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49c17a497980cd740b0d8a74998b831e8d742944c245c732c35c7d2fafb86827/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c831fa197eb040ab59649744e4fa44b632eda251171ae2c0653ba688edf1edd/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7aa1b80f-115e-4708-bc98-88d13ad66860/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:60 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b2dbc17-c725-4893-896e-4dd2ec2dbc13/volumes/kubernetes.io~projected/kube-api-access-826wq DeviceMajor:0 DeviceMinor:325 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-438 DeviceMajor:0 DeviceMinor:438 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-525 DeviceMajor:0 DeviceMinor:525 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b2f2c486-d661-4d59-981c-0b95a1f66fb4/volumes/kubernetes.io~projected/kube-api-access-sw4xv DeviceMajor:0 DeviceMinor:67 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/84bfdb37-6346-4384-8d97-d43aba6f03fe/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:208 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/550e690a-605e-47a4-ba95-74bcd26e0cc3/volumes/kubernetes.io~projected/kube-api-access-kk7l4 DeviceMajor:0 DeviceMinor:286 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd3b421a-9313-4f15-b81f-900717685fd6/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:205 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/928ed5956456ef15f9e219522ca60366bbf675bbaca85d60cb4ed63fd76dbbfa/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a32b60f2-3000-4fe8-a928-955de8c76c82/volumes/kubernetes.io~projected/kube-api-access-r6k4l DeviceMajor:0 DeviceMinor:65 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f839a6af4cc59cc154537387101d77f74429e09b52a630505d1737c8a234134f/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/226c41d5-de84-4950-978f-a1f0b4cbc319/volumes/kubernetes.io~projected/kube-api-access-67x9l DeviceMajor:0 DeviceMinor:228 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/84bfdb37-6346-4384-8d97-d43aba6f03fe/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:454 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3458688b-8142-492c-ab8b-bffec754c8c8/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:458 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dc529468b79e6b85d51b5854b713775940759b7a512c5c0ebacbe15b472060b1/userdata/shm DeviceMajor:0 DeviceMinor:372 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a32b60f2-3000-4fe8-a928-955de8c76c82/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/64086350-55f6-4da5-a871-3f23c1755371/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:59 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7aa1b80f-115e-4708-bc98-88d13ad66860/volumes/kubernetes.io~projected/kube-api-access-qvxrm DeviceMajor:0 DeviceMinor:63 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-513 DeviceMajor:0 DeviceMinor:513 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c65f9e423b79ea4d8c7c6522536eb48c1550c1bb189aad40fd355f8cd78429ef/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0cece9c-2008-49ce-87de-4804d4e7de08/volumes/kubernetes.io~projected/kube-api-access-j92gp DeviceMajor:0 DeviceMinor:221 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7140c65db18f03b4c55457261bcf97ff25e415df1bc664655ff2f0d85916a07e/userdata/shm DeviceMajor:0 DeviceMinor:466 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/11109f61-f4f0-4253-aacb-886c353a5772/volumes/kubernetes.io~projected/kube-api-access-vgccp DeviceMajor:0 DeviceMinor:227 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-484 DeviceMajor:0 DeviceMinor:484 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bf0fd8f-2294-4729-9f49-a151de8627aa/volumes/kubernetes.io~projected/kube-api-access-r2b29 DeviceMajor:0 DeviceMinor:45 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/84bfdb37-6346-4384-8d97-d43aba6f03fe/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:216 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ea7a7f39436bd95eb87aaa921bf467e4ccb983ecc7b9cb726badd8d5ea2b22e/userdata/shm DeviceMajor:0 DeviceMinor:341 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4d9c6673201363e110fb9941032ebf8c83fc19e1fe99fe2539d7b22f07db02c6/userdata/shm DeviceMajor:0 DeviceMinor:467 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2aca39e96a0a5359c3c6f3557b7fab773e71a0f6396b23c99359d1f48424db5d/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f020130a1364da047db4e8b77083514ccee215eae989a164c46e413faef9d03/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25c043a82f378ee1f768d77bc37682bee28ed6f94b374c58924c70d296371cda/userdata/shm DeviceMajor:0 DeviceMinor:523 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/163fe39a07496c2c1b0c6d137e35ddfe934972beac5729c3c172d8ceeb97c2d3/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/372b376c17a9437a2aaed178dc1027b6d2f6bd6e7a3642656f226e4ab82e94b3/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/807ca5626e858e6a12ee34390615264e14111f00424b2e4b7942a71865766972/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/13fd2c81-54ab-4afa-9afe-68094563bf08/volumes/kubernetes.io~projected/kube-api-access-vc5dn DeviceMajor:0 DeviceMinor:56 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ffa64e38-3f1b-49af-913f-139addefa359/volumes/kubernetes.io~projected/kube-api-access-5fgkx DeviceMajor:0 DeviceMinor:62 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11ba3d2b-2d12-458c-9984-1dd82c275e10/volumes/kubernetes.io~projected/kube-api-access-58wrk DeviceMajor:0 DeviceMinor:220 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/35a08208-7245-4e1b-9c0b-9e825a808365/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:226 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c59a8eea64518cccb015decbdeff9327a62f4223b713bc4278344fd1041ce992/userdata/shm DeviceMajor:0 DeviceMinor:339 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bbd26b51457e1453cc51ec773785a48957a5de44d7e58aaf6e47e64808180460/userdata/shm DeviceMajor:0 DeviceMinor:386 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ce930258-60d9-4988-913c-2f3cb78eac8d/volumes/kubernetes.io~projected/kube-api-access-v5gt6 DeviceMajor:0 DeviceMinor:456 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-492 DeviceMajor:0 DeviceMinor:492 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a32b60f2-3000-4fe8-a928-955de8c76c82/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/35a08208-7245-4e1b-9c0b-9e825a808365/volumes/kubernetes.io~projected/kube-api-access-h8258 DeviceMajor:0 DeviceMinor:229 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2/volumes/kubernetes.io~projected/kube-api-access-9hnzg DeviceMajor:0 DeviceMinor:356 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/95097323-8907-45bc-8967-0de718adefbd/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls DeviceMajor:0 DeviceMinor:449 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11ba3d2b-2d12-458c-9984-1dd82c275e10/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:203 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c0cece9c-2008-49ce-87de-4804d4e7de08/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:351 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-480 DeviceMajor:0 DeviceMinor:480 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-527 DeviceMajor:0 DeviceMinor:527 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:114ac894e52562d MacAddress:22:83:72:e3:52:ef Speed:10000 Mtu:8901} {Name:180790b45601d52 MacAddress:1a:e8:d4:13:98:c9 Speed:10000 Mtu:8901} {Name:25c043a82f378ee MacAddress:f6:56:ab:b5:c4:3b Speed:10000 Mtu:8901} {Name:2942f9ed8024d9e MacAddress:e6:f1:32:96:2f:d3 Speed:10000 Mtu:8901} {Name:2c831fa197eb040 MacAddress:02:a3:fa:f3:94:00 Speed:10000 Mtu:8901} {Name:373986040c020db MacAddress:8e:c3:66:c5:4d:8b Speed:10000 Mtu:8901} {Name:379969793eebc3c MacAddress:fe:ff:56:e5:70:85 Speed:10000 Mtu:8901} {Name:3d8709f5a2d9ef7 MacAddress:b6:f2:c4:19:aa:0f Speed:10000 Mtu:8901} {Name:49c17a497980cd7 MacAddress:66:6f:7e:88:61:39 Speed:10000 Mtu:8901} {Name:4d9c6673201363e MacAddress:d2:9d:69:08:e3:ea Speed:10000 Mtu:8901} {Name:4f020130a1364da MacAddress:b2:0c:a5:0a:b4:c1 Speed:10000 Mtu:8901} {Name:7140c65db18f03b MacAddress:c6:87:6f:e6:f8:b0 Speed:10000 Mtu:8901} {Name:72df4ed43b9eb34 MacAddress:6e:4d:10:a3:42:33 Speed:10000 Mtu:8901} {Name:9ea7a7f39436bd9 MacAddress:be:84:b8:53:df:9c Speed:10000 Mtu:8901} {Name:9f8402d180547bc MacAddress:0e:8c:16:f6:ca:19 Speed:10000 Mtu:8901} {Name:ab93116a47c2064 MacAddress:b2:a8:6e:eb:c8:19 Speed:10000 Mtu:8901} {Name:add64044c379823 MacAddress:c6:17:43:d7:f1:45 Speed:10000 Mtu:8901} {Name:b1aee542906e797 MacAddress:32:44:dc:9f:b7:b9 Speed:10000 Mtu:8901} {Name:b6a2dd3b149a596 MacAddress:9e:1f:53:12:86:08 Speed:10000 Mtu:8901} {Name:bbd26b51457e145 MacAddress:2a:08:93:bc:c2:5b Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:e4:5d:83:d7 Speed:0 Mtu:9001} {Name:br-int MacAddress:96:78:b0:d5:65:2d Speed:0 Mtu:8901} {Name:c2fab8c8f944259 MacAddress:2e:72:a6:2e:ae:57 Speed:10000 Mtu:8901} {Name:c59a8eea64518cc MacAddress:ca:ae:5d:27:35:ac Speed:10000 Mtu:8901} {Name:d6c5d5133bfcf2a MacAddress:f2:77:75:4e:7e:32 Speed:10000 Mtu:8901} {Name:dc529468b79e6b8 MacAddress:0e:84:00:2e:b7:5a Speed:10000 Mtu:8901} {Name:e3fff7abd1c2d54 MacAddress:f6:03:45:6d:74:ad Speed:10000 Mtu:8901} {Name:eab77923780ae46 MacAddress:b6:b4:a2:18:aa:7b Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:e4:5d:83:d7 Speed:0 Mtu:9001} {Name:f839a6af4cc59cc MacAddress:da:5c:e1:6e:b4:1d Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:96:ad:66:d3:78:18 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:aa:4d:ed:24:92:8e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:49:57.533722 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.533726 6006 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:49:57.534105 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.533776 6006 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:49:57.534105 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.534070 6006 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:49:57.534236 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.534093 6006 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-7.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:49:57.534374 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.534239 6006 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:49:57.534374 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.534263 6006 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:49:57.534374 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.534278 6006 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:49:57.534374 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.534298 6006 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:49:57.534374 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.534319 6006 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:49:57.534536 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.534483 6006 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:49:57.534603 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.534593 6006 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:49:57.534641 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.534630 6006 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:49:57.534641 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.534641 6006 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:49:57.534735 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.534655 6006 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:49:57.534735 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.534664 6006 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:49:57.534735 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.534672 6006 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:49:57.535506 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.535488 6006 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:49:57.535506 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.535503 6006 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:49:57.536091 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.536062 6006 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:49:57.551374 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.551355 6006 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:49:57.551374 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.551370 6006 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:49:57.551511 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.551483 6006 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 21:49:57.551697 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.551685 6006 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:49:57.551728 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.551715 6006 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:49:57.551839 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.551830 6006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:49:57.551872 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.551845 6006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:49:57.551872 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.551851 6006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:49:57.551872 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.551857 6006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:49:57.551951 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.551874 6006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:49:57.551951 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.551880 6006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:49:57.551951 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.551886 6006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:49:57.551951 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.551891 6006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:49:57.551951 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.551899 6006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:49:57.551951 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.551905 6006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:49:57.551951 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.551913 6006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:49:57.551951 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.551922 6006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:49:57.551951 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.551949 6006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:49:57.551951 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.551955 6006 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:49:57.556322 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.556300 6006 server.go:1263] "Started kubelet" Oct 30 21:49:57.556477 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.556375 6006 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:49:57.556794 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.556737 6006 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:49:57.556916 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.556816 6006 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:49:57.556859 ip-10-0-135-7 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:49:57.557724 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.557704 6006 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:49:57.558205 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.558182 6006 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:49:57.561355 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.560646 6006 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:49:57.561355 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.560659 6006 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:49:57.561355 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.560779 6006 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:49:57.561355 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.560743 6006 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:43:46 +0000 UTC" deadline="2028-05-24 21:41:54.508576819 +0000 UTC" Oct 30 21:49:57.561355 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.560799 6006 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:49:57.561355 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.560803 6006 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13727h51m56.947778087s" Oct 30 21:49:57.561355 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.560791 6006 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:49:57.561355 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:49:57.560832 6006 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-7.us-west-2.compute.internal\" not found" Oct 30 21:49:57.565155 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.565111 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3458688b-8142-492c-ab8b-bffec754c8c8" volumeName="kubernetes.io/configmap/3458688b-8142-492c-ab8b-bffec754c8c8-registry-certificates" seLinuxMountContext="" Oct 30 21:49:57.565239 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.565159 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b2dbc17-c725-4893-896e-4dd2ec2dbc13" volumeName="kubernetes.io/configmap/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 21:49:57.565239 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.565177 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51" volumeName="kubernetes.io/configmap/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51-trusted-ca" seLinuxMountContext="" Oct 30 21:49:57.565239 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.565193 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51" volumeName="kubernetes.io/projected/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51-kube-api-access-lkjjr" seLinuxMountContext="" Oct 30 21:49:57.565239 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.565207 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64086350-55f6-4da5-a871-3f23c1755371" volumeName="kubernetes.io/secret/64086350-55f6-4da5-a871-3f23c1755371-agent-certs" seLinuxMountContext="" Oct 30 21:49:57.565239 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.565219 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84bfdb37-6346-4384-8d97-d43aba6f03fe" volumeName="kubernetes.io/empty-dir/84bfdb37-6346-4384-8d97-d43aba6f03fe-ca-trust-extracted" seLinuxMountContext="" Oct 30 21:49:57.565239 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.565230 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9161cdbd-c44f-4a8c-b918-107fc7001f3a" volumeName="kubernetes.io/projected/9161cdbd-c44f-4a8c-b918-107fc7001f3a-kube-api-access-9hz67" seLinuxMountContext="" Oct 30 21:49:57.565489 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.565244 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a32b60f2-3000-4fe8-a928-955de8c76c82" volumeName="kubernetes.io/projected/a32b60f2-3000-4fe8-a928-955de8c76c82-kube-api-access-r6k4l" seLinuxMountContext="" Oct 30 21:49:57.566282 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.565763 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13fd2c81-54ab-4afa-9afe-68094563bf08" volumeName="kubernetes.io/configmap/13fd2c81-54ab-4afa-9afe-68094563bf08-cni-binary-copy" seLinuxMountContext="" Oct 30 21:49:57.566282 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.565807 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35a08208-7245-4e1b-9c0b-9e825a808365" volumeName="kubernetes.io/projected/35a08208-7245-4e1b-9c0b-9e825a808365-kube-api-access-h8258" seLinuxMountContext="" Oct 30 21:49:57.566282 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.565837 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9df43c88-a01f-4ec7-a385-d5afcf6f9e6c" volumeName="kubernetes.io/projected/9df43c88-a01f-4ec7-a385-d5afcf6f9e6c-kube-api-access-thhdq" seLinuxMountContext="" Oct 30 21:49:57.566282 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.565861 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a32b60f2-3000-4fe8-a928-955de8c76c82" volumeName="kubernetes.io/empty-dir/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-tuned" seLinuxMountContext="" Oct 30 21:49:57.566282 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.565880 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0cece9c-2008-49ce-87de-4804d4e7de08" volumeName="kubernetes.io/configmap/c0cece9c-2008-49ce-87de-4804d4e7de08-service-ca-bundle" seLinuxMountContext="" Oct 30 21:49:57.566282 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.565900 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc3ff956-6466-46e3-b734-13a3ac148d36" volumeName="kubernetes.io/projected/fc3ff956-6466-46e3-b734-13a3ac148d36-kube-api-access-k9lcg" seLinuxMountContext="" Oct 30 21:49:57.566282 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.565927 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffa64e38-3f1b-49af-913f-139addefa359" volumeName="kubernetes.io/projected/ffa64e38-3f1b-49af-913f-139addefa359-kube-api-access-5fgkx" seLinuxMountContext="" Oct 30 21:49:57.566282 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.565942 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11109f61-f4f0-4253-aacb-886c353a5772" volumeName="kubernetes.io/projected/11109f61-f4f0-4253-aacb-886c353a5772-kube-api-access-vgccp" seLinuxMountContext="" Oct 30 21:49:57.566282 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.565960 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3" volumeName="kubernetes.io/secret/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-metrics-tls" seLinuxMountContext="" Oct 30 21:49:57.566282 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.565975 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84bfdb37-6346-4384-8d97-d43aba6f03fe" volumeName="kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls" seLinuxMountContext="" Oct 30 21:49:57.566282 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.565987 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc3ff956-6466-46e3-b734-13a3ac148d36" volumeName="kubernetes.io/configmap/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-config" seLinuxMountContext="" Oct 30 21:49:57.566282 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.566005 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13fd2c81-54ab-4afa-9afe-68094563bf08" volumeName="kubernetes.io/configmap/13fd2c81-54ab-4afa-9afe-68094563bf08-multus-daemon-config" seLinuxMountContext="" Oct 30 21:49:57.566282 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.566018 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35a08208-7245-4e1b-9c0b-9e825a808365" volumeName="kubernetes.io/secret/35a08208-7245-4e1b-9c0b-9e825a808365-serving-cert" seLinuxMountContext="" Oct 30 21:49:57.566282 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.566036 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47917159-29d7-45b9-bbf2-13cef4408de9" volumeName="kubernetes.io/secret/47917159-29d7-45b9-bbf2-13cef4408de9-networking-console-plugin-cert" seLinuxMountContext="" Oct 30 21:49:57.566282 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.566051 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4" volumeName="kubernetes.io/projected/1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4-kube-api-access-7q7c2" seLinuxMountContext="" Oct 30 21:49:57.566282 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.566064 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dce5b38-e1a7-41e3-8100-96af4e3c2b06" volumeName="kubernetes.io/projected/0dce5b38-e1a7-41e3-8100-96af4e3c2b06-kube-api-access-jr7zd" seLinuxMountContext="" Oct 30 21:49:57.566282 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.566081 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e73e56e-9619-4b97-bf20-a6291acf6355" volumeName="kubernetes.io/projected/6e73e56e-9619-4b97-bf20-a6291acf6355-kube-api-access-f54bb" seLinuxMountContext="" Oct 30 21:49:57.566282 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.566095 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95097323-8907-45bc-8967-0de718adefbd" volumeName="kubernetes.io/configmap/95097323-8907-45bc-8967-0de718adefbd-telemetry-config" seLinuxMountContext="" Oct 30 21:49:57.566282 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.566114 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd3b421a-9313-4f15-b81f-900717685fd6" volumeName="kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs" seLinuxMountContext="" Oct 30 21:49:57.566282 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.566130 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7fe5efb-7029-41b8-8bbe-f37f49410f27" volumeName="kubernetes.io/projected/c7fe5efb-7029-41b8-8bbe-f37f49410f27-kube-api-access-fn7gh" seLinuxMountContext="" Oct 30 21:49:57.566282 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.566158 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2" volumeName="kubernetes.io/configmap/d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2-signing-cabundle" seLinuxMountContext="" Oct 30 21:49:57.566282 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.566171 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3" volumeName="kubernetes.io/projected/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-kube-api-access-hgd6c" seLinuxMountContext="" Oct 30 21:49:57.566282 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.566190 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95097323-8907-45bc-8967-0de718adefbd" volumeName="kubernetes.io/projected/95097323-8907-45bc-8967-0de718adefbd-kube-api-access-j58ln" seLinuxMountContext="" Oct 30 21:49:57.566282 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.566205 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd3b421a-9313-4f15-b81f-900717685fd6" volumeName="kubernetes.io/projected/bd3b421a-9313-4f15-b81f-900717685fd6-kube-api-access-j2qx6" seLinuxMountContext="" Oct 30 21:49:57.566282 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.566238 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ceedf44e-88c1-49cf-901e-75558703c361" volumeName="kubernetes.io/configmap/ceedf44e-88c1-49cf-901e-75558703c361-serviceca" seLinuxMountContext="" Oct 30 21:49:57.567369 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.566384 6006 factory.go:55] Registering systemd factory Oct 30 21:49:57.567369 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.566417 6006 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:49:57.567369 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.566847 6006 factory.go:153] Registering CRI-O factory Oct 30 21:49:57.567369 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.566866 6006 factory.go:223] Registration of the crio container factory successfully Oct 30 21:49:57.567369 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.566933 6006 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:49:57.567369 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.566952 6006 factory.go:103] Registering Raw factory Oct 30 21:49:57.567369 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.567026 6006 manager.go:1199] Started watching for new ooms in manager Oct 30 21:49:57.568411 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.568394 6006 manager.go:319] Starting recovery of all containers Oct 30 21:49:57.571154 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.569900 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f71e935d-f091-4f16-820b-8d7fc632f003" volumeName="kubernetes.io/projected/f71e935d-f091-4f16-820b-8d7fc632f003-kube-api-access-vdhdv" seLinuxMountContext="" Oct 30 21:49:57.571154 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.570592 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffa64e38-3f1b-49af-913f-139addefa359" volumeName="kubernetes.io/configmap/ffa64e38-3f1b-49af-913f-139addefa359-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 21:49:57.571154 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.570614 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3458688b-8142-492c-ab8b-bffec754c8c8" volumeName="kubernetes.io/secret/3458688b-8142-492c-ab8b-bffec754c8c8-installation-pull-secrets" seLinuxMountContext="" Oct 30 21:49:57.571154 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.570634 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64086350-55f6-4da5-a871-3f23c1755371" volumeName="kubernetes.io/configmap/64086350-55f6-4da5-a871-3f23c1755371-konnectivity-ca" seLinuxMountContext="" Oct 30 21:49:57.571154 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.570677 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7aa1b80f-115e-4708-bc98-88d13ad66860" volumeName="kubernetes.io/projected/7aa1b80f-115e-4708-bc98-88d13ad66860-kube-api-access-qvxrm" seLinuxMountContext="" Oct 30 21:49:57.571154 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.570695 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84bfdb37-6346-4384-8d97-d43aba6f03fe" volumeName="kubernetes.io/secret/84bfdb37-6346-4384-8d97-d43aba6f03fe-installation-pull-secrets" seLinuxMountContext="" Oct 30 21:49:57.571154 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.570712 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b60900b-7a0e-4039-97af-c8157ae19ca0" volumeName="kubernetes.io/empty-dir/8b60900b-7a0e-4039-97af-c8157ae19ca0-archives-path" seLinuxMountContext="" Oct 30 21:49:57.571154 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.570800 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0cece9c-2008-49ce-87de-4804d4e7de08" volumeName="kubernetes.io/secret/c0cece9c-2008-49ce-87de-4804d4e7de08-serving-cert" seLinuxMountContext="" Oct 30 21:49:57.571154 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.570834 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11ba3d2b-2d12-458c-9984-1dd82c275e10" volumeName="kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-default-certificate" seLinuxMountContext="" Oct 30 21:49:57.571154 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.570850 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="226c41d5-de84-4950-978f-a1f0b4cbc319" volumeName="kubernetes.io/projected/226c41d5-de84-4950-978f-a1f0b4cbc319-kube-api-access-67x9l" seLinuxMountContext="" Oct 30 21:49:57.571154 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.570875 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3458688b-8142-492c-ab8b-bffec754c8c8" volumeName="kubernetes.io/projected/3458688b-8142-492c-ab8b-bffec754c8c8-kube-api-access-hxnj9" seLinuxMountContext="" Oct 30 21:49:57.571154 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.570890 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3458688b-8142-492c-ab8b-bffec754c8c8" volumeName="kubernetes.io/projected/3458688b-8142-492c-ab8b-bffec754c8c8-bound-sa-token" seLinuxMountContext="" Oct 30 21:49:57.571154 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.570907 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3458688b-8142-492c-ab8b-bffec754c8c8" volumeName="kubernetes.io/secret/3458688b-8142-492c-ab8b-bffec754c8c8-image-registry-private-configuration" seLinuxMountContext="" Oct 30 21:49:57.571154 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.570953 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84bfdb37-6346-4384-8d97-d43aba6f03fe" volumeName="kubernetes.io/configmap/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-certificates" seLinuxMountContext="" Oct 30 21:49:57.571154 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.570999 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84bfdb37-6346-4384-8d97-d43aba6f03fe" volumeName="kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-bound-sa-token" seLinuxMountContext="" Oct 30 21:49:57.571154 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.571028 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0cece9c-2008-49ce-87de-4804d4e7de08" volumeName="kubernetes.io/configmap/c0cece9c-2008-49ce-87de-4804d4e7de08-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:49:57.571154 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.571074 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35a08208-7245-4e1b-9c0b-9e825a808365" volumeName="kubernetes.io/configmap/35a08208-7245-4e1b-9c0b-9e825a808365-config" seLinuxMountContext="" Oct 30 21:49:57.571154 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.571103 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51" volumeName="kubernetes.io/configmap/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51-config" seLinuxMountContext="" Oct 30 21:49:57.571154 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.571153 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b60900b-7a0e-4039-97af-c8157ae19ca0" volumeName="kubernetes.io/projected/8b60900b-7a0e-4039-97af-c8157ae19ca0-kube-api-access-fcm8z" seLinuxMountContext="" Oct 30 21:49:57.571928 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.571205 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0cece9c-2008-49ce-87de-4804d4e7de08" volumeName="kubernetes.io/empty-dir/c0cece9c-2008-49ce-87de-4804d4e7de08-tmp" seLinuxMountContext="" Oct 30 21:49:57.571928 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.571304 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7fe5efb-7029-41b8-8bbe-f37f49410f27" volumeName="kubernetes.io/secret/c7fe5efb-7029-41b8-8bbe-f37f49410f27-cert" seLinuxMountContext="" Oct 30 21:49:57.571928 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.571321 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2" volumeName="kubernetes.io/projected/d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2-kube-api-access-9hnzg" seLinuxMountContext="" Oct 30 21:49:57.571928 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.571355 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2" volumeName="kubernetes.io/secret/d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2-signing-key" seLinuxMountContext="" Oct 30 21:49:57.571928 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.571424 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffa64e38-3f1b-49af-913f-139addefa359" volumeName="kubernetes.io/configmap/ffa64e38-3f1b-49af-913f-139addefa359-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 21:49:57.571928 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.571438 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11109f61-f4f0-4253-aacb-886c353a5772" volumeName="kubernetes.io/configmap/11109f61-f4f0-4253-aacb-886c353a5772-config" seLinuxMountContext="" Oct 30 21:49:57.571928 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.571457 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7aa1b80f-115e-4708-bc98-88d13ad66860" volumeName="kubernetes.io/configmap/7aa1b80f-115e-4708-bc98-88d13ad66860-ovnkube-script-lib" seLinuxMountContext="" Oct 30 21:49:57.571928 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.571480 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bf0fd8f-2294-4729-9f49-a151de8627aa" volumeName="kubernetes.io/projected/8bf0fd8f-2294-4729-9f49-a151de8627aa-kube-api-access-r2b29" seLinuxMountContext="" Oct 30 21:49:57.571928 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.571589 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11ba3d2b-2d12-458c-9984-1dd82c275e10" volumeName="kubernetes.io/configmap/11ba3d2b-2d12-458c-9984-1dd82c275e10-service-ca-bundle" seLinuxMountContext="" Oct 30 21:49:57.571928 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.571605 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13fd2c81-54ab-4afa-9afe-68094563bf08" volumeName="kubernetes.io/projected/13fd2c81-54ab-4afa-9afe-68094563bf08-kube-api-access-vc5dn" seLinuxMountContext="" Oct 30 21:49:57.571928 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.571618 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4579f026-39bd-465c-b0db-41676370ba55" volumeName="kubernetes.io/projected/4579f026-39bd-465c-b0db-41676370ba55-kube-api-access-l8ctq" seLinuxMountContext="" Oct 30 21:49:57.571928 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.571628 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7aa1b80f-115e-4708-bc98-88d13ad66860" volumeName="kubernetes.io/secret/7aa1b80f-115e-4708-bc98-88d13ad66860-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 21:49:57.571928 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.571638 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b60900b-7a0e-4039-97af-c8157ae19ca0" volumeName="kubernetes.io/configmap/8b60900b-7a0e-4039-97af-c8157ae19ca0-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:49:57.571928 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.571649 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b60900b-7a0e-4039-97af-c8157ae19ca0" volumeName="kubernetes.io/configmap/8b60900b-7a0e-4039-97af-c8157ae19ca0-service-ca-bundle" seLinuxMountContext="" Oct 30 21:49:57.571928 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.571658 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0cece9c-2008-49ce-87de-4804d4e7de08" volumeName="kubernetes.io/empty-dir/c0cece9c-2008-49ce-87de-4804d4e7de08-snapshots" seLinuxMountContext="" Oct 30 21:49:57.571928 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.571914 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc3ff956-6466-46e3-b734-13a3ac148d36" volumeName="kubernetes.io/secret/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-tls" seLinuxMountContext="" Oct 30 21:49:57.572424 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.571954 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11ba3d2b-2d12-458c-9984-1dd82c275e10" volumeName="kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-stats-auth" seLinuxMountContext="" Oct 30 21:49:57.572424 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.571978 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3" volumeName="kubernetes.io/empty-dir/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-tmp-dir" seLinuxMountContext="" Oct 30 21:49:57.572424 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.572009 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2f2c486-d661-4d59-981c-0b95a1f66fb4" volumeName="kubernetes.io/projected/b2f2c486-d661-4d59-981c-0b95a1f66fb4-kube-api-access-sw4xv" seLinuxMountContext="" Oct 30 21:49:57.572424 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.572026 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce930258-60d9-4988-913c-2f3cb78eac8d" volumeName="kubernetes.io/projected/ce930258-60d9-4988-913c-2f3cb78eac8d-kube-api-access-v5gt6" seLinuxMountContext="" Oct 30 21:49:57.572424 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.572052 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ceedf44e-88c1-49cf-901e-75558703c361" volumeName="kubernetes.io/projected/ceedf44e-88c1-49cf-901e-75558703c361-kube-api-access-s8pl2" seLinuxMountContext="" Oct 30 21:49:57.572424 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.572072 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11109f61-f4f0-4253-aacb-886c353a5772" volumeName="kubernetes.io/secret/11109f61-f4f0-4253-aacb-886c353a5772-serving-cert" seLinuxMountContext="" Oct 30 21:49:57.572424 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.572090 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d430f7e-00f1-4b9e-9427-3e9971cfff31" volumeName="kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret" seLinuxMountContext="" Oct 30 21:49:57.572424 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.572108 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3458688b-8142-492c-ab8b-bffec754c8c8" volumeName="kubernetes.io/configmap/3458688b-8142-492c-ab8b-bffec754c8c8-trusted-ca" seLinuxMountContext="" Oct 30 21:49:57.572424 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.572124 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3458688b-8142-492c-ab8b-bffec754c8c8" volumeName="kubernetes.io/projected/3458688b-8142-492c-ab8b-bffec754c8c8-registry-tls" seLinuxMountContext="" Oct 30 21:49:57.572424 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.572147 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3" volumeName="kubernetes.io/configmap/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-config-volume" seLinuxMountContext="" Oct 30 21:49:57.572424 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.572179 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b2dbc17-c725-4893-896e-4dd2ec2dbc13" volumeName="kubernetes.io/empty-dir/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-data-volume" seLinuxMountContext="" Oct 30 21:49:57.572424 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.572198 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7aa1b80f-115e-4708-bc98-88d13ad66860" volumeName="kubernetes.io/configmap/7aa1b80f-115e-4708-bc98-88d13ad66860-ovnkube-config" seLinuxMountContext="" Oct 30 21:49:57.572424 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.572218 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7aa1b80f-115e-4708-bc98-88d13ad66860" volumeName="kubernetes.io/configmap/7aa1b80f-115e-4708-bc98-88d13ad66860-env-overrides" seLinuxMountContext="" Oct 30 21:49:57.572424 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.572233 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3458688b-8142-492c-ab8b-bffec754c8c8" volumeName="kubernetes.io/empty-dir/3458688b-8142-492c-ab8b-bffec754c8c8-ca-trust-extracted" seLinuxMountContext="" Oct 30 21:49:57.572424 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.572272 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="550e690a-605e-47a4-ba95-74bcd26e0cc3" volumeName="kubernetes.io/projected/550e690a-605e-47a4-ba95-74bcd26e0cc3-kube-api-access-kk7l4" seLinuxMountContext="" Oct 30 21:49:57.572424 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.572289 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84bfdb37-6346-4384-8d97-d43aba6f03fe" volumeName="kubernetes.io/configmap/84bfdb37-6346-4384-8d97-d43aba6f03fe-trusted-ca" seLinuxMountContext="" Oct 30 21:49:57.572424 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.572308 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84bfdb37-6346-4384-8d97-d43aba6f03fe" volumeName="kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-kube-api-access-vss2v" seLinuxMountContext="" Oct 30 21:49:57.572424 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.572327 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8df7e7fd-be92-4d37-af33-930b7b36cbe6" volumeName="kubernetes.io/secret/8df7e7fd-be92-4d37-af33-930b7b36cbe6-tls-certificates" seLinuxMountContext="" Oct 30 21:49:57.572424 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.572362 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9161cdbd-c44f-4a8c-b918-107fc7001f3a" volumeName="kubernetes.io/configmap/9161cdbd-c44f-4a8c-b918-107fc7001f3a-iptables-alerter-script" seLinuxMountContext="" Oct 30 21:49:57.572424 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.572381 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a32b60f2-3000-4fe8-a928-955de8c76c82" volumeName="kubernetes.io/empty-dir/a32b60f2-3000-4fe8-a928-955de8c76c82-tmp" seLinuxMountContext="" Oct 30 21:49:57.572424 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.572413 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0cece9c-2008-49ce-87de-4804d4e7de08" volumeName="kubernetes.io/projected/c0cece9c-2008-49ce-87de-4804d4e7de08-kube-api-access-j92gp" seLinuxMountContext="" Oct 30 21:49:57.573443 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.572437 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11ba3d2b-2d12-458c-9984-1dd82c275e10" volumeName="kubernetes.io/projected/11ba3d2b-2d12-458c-9984-1dd82c275e10-kube-api-access-58wrk" seLinuxMountContext="" Oct 30 21:49:57.573443 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.572452 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11ba3d2b-2d12-458c-9984-1dd82c275e10" volumeName="kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-metrics-certs" seLinuxMountContext="" Oct 30 21:49:57.573443 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.572499 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4" volumeName="kubernetes.io/empty-dir/1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4-tmp-dir" seLinuxMountContext="" Oct 30 21:49:57.573443 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.572532 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b2dbc17-c725-4893-896e-4dd2ec2dbc13" volumeName="kubernetes.io/secret/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 21:49:57.573443 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.572548 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84bfdb37-6346-4384-8d97-d43aba6f03fe" volumeName="kubernetes.io/secret/84bfdb37-6346-4384-8d97-d43aba6f03fe-image-registry-private-configuration" seLinuxMountContext="" Oct 30 21:49:57.576636 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.576297 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffa64e38-3f1b-49af-913f-139addefa359" volumeName="kubernetes.io/configmap/ffa64e38-3f1b-49af-913f-139addefa359-cni-binary-copy" seLinuxMountContext="" Oct 30 21:49:57.576636 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.576375 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47917159-29d7-45b9-bbf2-13cef4408de9" volumeName="kubernetes.io/configmap/47917159-29d7-45b9-bbf2-13cef4408de9-nginx-conf" seLinuxMountContext="" Oct 30 21:49:57.576636 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.576391 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b2dbc17-c725-4893-896e-4dd2ec2dbc13" volumeName="kubernetes.io/projected/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-kube-api-access-826wq" seLinuxMountContext="" Oct 30 21:49:57.576636 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.576421 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51" volumeName="kubernetes.io/secret/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51-serving-cert" seLinuxMountContext="" Oct 30 21:49:57.576636 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.576434 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95097323-8907-45bc-8967-0de718adefbd" volumeName="kubernetes.io/secret/95097323-8907-45bc-8967-0de718adefbd-cluster-monitoring-operator-tls" seLinuxMountContext="" Oct 30 21:49:57.576636 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.576446 6006 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce930258-60d9-4988-913c-2f3cb78eac8d" volumeName="kubernetes.io/empty-dir/ce930258-60d9-4988-913c-2f3cb78eac8d-tmp" seLinuxMountContext="" Oct 30 21:49:57.576636 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.576459 6006 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:49:57.576636 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.576469 6006 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:49:57.583501 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.583481 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 21:49:57.583989 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.583963 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 21:49:57.597701 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.593724 6006 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:49:57.597701 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.595214 6006 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:49:57.597701 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.595241 6006 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:49:57.597701 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.595365 6006 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:49:57.597701 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:49:57.595439 6006 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 21:49:57.621764 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.621734 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 21:49:57.621942 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.621773 6006 generic.go:356] "Generic (PLEG): container finished" podID="5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51" containerID="b131ed16214c61457d42c061278f83843746a5a6736eb8f5a8d2ee4ddf210a2a" exitCode=255 Oct 30 21:49:57.649603 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.649569 6006 generic.go:356] "Generic (PLEG): container finished" podID="ffa64e38-3f1b-49af-913f-139addefa359" containerID="22144b84ceaa81b5b23c27729b939d9027ca8acc4583ef62e56fd254c585a374" exitCode=0 Oct 30 21:49:57.649603 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.649598 6006 generic.go:356] "Generic (PLEG): container finished" podID="ffa64e38-3f1b-49af-913f-139addefa359" containerID="fb417171a083de4a68b44d1fca299d43787122ac6e2692bceb3fc060dd630155" exitCode=0 Oct 30 21:49:57.649603 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.649607 6006 generic.go:356] "Generic (PLEG): container finished" podID="ffa64e38-3f1b-49af-913f-139addefa359" containerID="7a0096a9566e03670dfe5fa22ee26965fed73e5678cfca8d098fb29a8013d6d0" exitCode=0 Oct 30 21:49:57.649843 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.649614 6006 generic.go:356] "Generic (PLEG): container finished" podID="ffa64e38-3f1b-49af-913f-139addefa359" containerID="101d52438331cc44886088fab89f5f41da997f928a88e1c3dd3da632a9b73f49" exitCode=0 Oct 30 21:49:57.649843 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.649622 6006 generic.go:356] "Generic (PLEG): container finished" podID="ffa64e38-3f1b-49af-913f-139addefa359" containerID="8a9cc2cff1cd2dbeedaee92b373dc530a85e2120851aaa05532ed00da25336f1" exitCode=0 Oct 30 21:49:57.649843 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.649629 6006 generic.go:356] "Generic (PLEG): container finished" podID="ffa64e38-3f1b-49af-913f-139addefa359" containerID="ef7b822b982cd0f1a5bc539eadafc7b0cb6e0d6e3b4057e6de4588839bce0d7f" exitCode=0 Oct 30 21:49:57.660444 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.660408 6006 generic.go:356] "Generic (PLEG): container finished" podID="556c8f6eb09b355e5322f215128e2cca" containerID="78c0e6ae83bcc50b9009d1b177396980329d8d01b22ae1f7ab4170f20b29d6eb" exitCode=0 Oct 30 21:49:57.660913 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:49:57.660890 6006 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-7.us-west-2.compute.internal\" not found" Oct 30 21:49:57.691516 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.691482 6006 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:49:57.695519 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:49:57.695494 6006 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 21:49:57.698055 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.698038 6006 manager.go:324] Recovery completed Oct 30 21:49:57.723422 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.723396 6006 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:49:57.725969 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.725947 6006 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:49:57.726071 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.725981 6006 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:49:57.726071 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.725991 6006 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:49:57.729478 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.729454 6006 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:49:57.729478 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.729476 6006 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:49:57.729620 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.729536 6006 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:49:57.729620 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.729477 6006 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:49:57.729770 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.729754 6006 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 21:49:57.729809 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.729770 6006 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 21:49:57.729809 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.729791 6006 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 21:49:57.729809 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.729797 6006 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 21:49:57.729809 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.729803 6006 policy_none.go:50] "Start" Oct 30 21:49:57.729931 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.729813 6006 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:49:57.729931 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.729824 6006 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:49:57.730042 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.729953 6006 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 21:49:57.730042 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.729963 6006 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 21:49:57.730042 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.729969 6006 policy_none.go:44] "Start" Oct 30 21:49:57.730607 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.730586 6006 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:49:57.730691 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.730623 6006 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:49:57.730691 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.730637 6006 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:49:57.735038 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.735011 6006 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:49:57.739720 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.739704 6006 manager.go:341] "Starting Device Plugin manager" Oct 30 21:49:57.739814 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:49:57.739735 6006 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:49:57.739814 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.739746 6006 server.go:87] "Starting device plugin registration server" Oct 30 21:49:57.740078 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.740066 6006 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:49:57.740599 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.740079 6006 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:49:57.740674 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.740628 6006 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:49:57.740761 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.740726 6006 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:49:57.740813 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.740774 6006 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:49:57.740813 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.740781 6006 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:49:57.740908 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.740833 6006 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:49:57.740759664Z","UUID":"67c8b1fc-0fdc-41ef-8d84-4881499bbb2a","Handler":null,"Name":"","Endpoint":""} Oct 30 21:49:57.742777 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:49:57.742753 6006 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:49:57.742777 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.742770 6006 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:49:57.742943 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:49:57.742799 6006 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-7.us-west-2.compute.internal\" not found" Oct 30 21:49:57.742943 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.742803 6006 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:49:57.773676 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.773597 6006 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:49:57.780721 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.780692 6006 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:49:57.782958 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.782937 6006 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 21:49:57.829897 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.829852 6006 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:49:57.841081 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.841055 6006 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:49:57.842181 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.842162 6006 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:49:57.842263 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.842191 6006 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:49:57.842263 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.842202 6006 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-7.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:49:57.842363 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.842311 6006 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:49:57.896601 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.896547 6006 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal"] Oct 30 21:49:57.897323 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.897295 6006 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:49:57.897469 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.897282 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal" event={"ID":"774172d5b7cfcf0d4b41c09fc8c830a1","Type":"ContainerStarted","Data":"b32a86641040e3fc823c6e0e52cf588c29bbbe4d667949d083a0f7addd3979eb"} Oct 30 21:49:57.897469 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.897416 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal" event={"ID":"774172d5b7cfcf0d4b41c09fc8c830a1","Type":"ContainerStarted","Data":"807ca5626e858e6a12ee34390615264e14111f00424b2e4b7942a71865766972"} Oct 30 21:49:57.897536 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.897483 6006 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:49:57.897536 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.897502 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal" event={"ID":"556c8f6eb09b355e5322f215128e2cca","Type":"ContainerStarted","Data":"73dbff0e55bccca4e361c26c33d1a7d0eaaeea76cbb8c0193f2243323dd31372"} Oct 30 21:49:57.897536 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.897518 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal" event={"ID":"556c8f6eb09b355e5322f215128e2cca","Type":"ContainerDied","Data":"78c0e6ae83bcc50b9009d1b177396980329d8d01b22ae1f7ab4170f20b29d6eb"} Oct 30 21:49:57.897536 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.897532 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal" event={"ID":"556c8f6eb09b355e5322f215128e2cca","Type":"ContainerStarted","Data":"a8f41036b33b5ee78d7094694eb5bbd534b732f7cc74ed189e023f1179486594"} Oct 30 21:49:57.946173 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.946142 6006 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:49:57.946324 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.946223 6006 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:49:57.959458 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.959429 6006 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:49:57.959587 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:49:57.959481 6006 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:49:57.960106 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.960090 6006 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:49:57.960145 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:49:57.960135 6006 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:49:57.983372 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.983344 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/774172d5b7cfcf0d4b41c09fc8c830a1-config\") pod \"kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal\" (UID: \"774172d5b7cfcf0d4b41c09fc8c830a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:49:57.983478 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.983396 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/556c8f6eb09b355e5322f215128e2cca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal\" (UID: \"556c8f6eb09b355e5322f215128e2cca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:49:57.983478 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:57.983424 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/556c8f6eb09b355e5322f215128e2cca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal\" (UID: \"556c8f6eb09b355e5322f215128e2cca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:49:58.084123 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.084088 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/556c8f6eb09b355e5322f215128e2cca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal\" (UID: \"556c8f6eb09b355e5322f215128e2cca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:49:58.084123 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.084131 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/774172d5b7cfcf0d4b41c09fc8c830a1-config\") pod \"kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal\" (UID: \"774172d5b7cfcf0d4b41c09fc8c830a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:49:58.084380 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.084166 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/556c8f6eb09b355e5322f215128e2cca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal\" (UID: \"556c8f6eb09b355e5322f215128e2cca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:49:58.084380 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.084207 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/556c8f6eb09b355e5322f215128e2cca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal\" (UID: \"556c8f6eb09b355e5322f215128e2cca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:49:58.084380 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.084226 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/774172d5b7cfcf0d4b41c09fc8c830a1-config\") pod \"kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal\" (UID: \"774172d5b7cfcf0d4b41c09fc8c830a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:49:58.084380 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.084300 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/556c8f6eb09b355e5322f215128e2cca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal\" (UID: \"556c8f6eb09b355e5322f215128e2cca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:49:58.535091 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.535006 6006 apiserver.go:52] "Watching apiserver" Oct 30 21:49:58.634927 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.634895 6006 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:49:58.635731 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.635697 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl","openshift-network-operator/iptables-alerter-gft82","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-55pjz","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p54hn","openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bc452","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal","openshift-console/downloads-6dd789c8b7-tqt4b","kube-system/global-pull-secret-syncer-lfq7z","openshift-multus/network-metrics-daemon-69j9v","openshift-network-diagnostics/network-check-source-6f78d59f99-989z4","kube-system/kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal","openshift-network-console/networking-console-plugin-86c478bb88-5xd2z","openshift-ingress/router-default-76684dbbf-8mwh7","openshift-multus/multus-mmdsp","openshift-image-registry/node-ca-x4655","openshift-insights/insights-runtime-extractor-hkqxb","openshift-dns/node-resolver-74zqz","openshift-console-operator/console-operator-7cd56b8dbc-8w8b5","openshift-image-registry/image-registry-6c68f8f5c6-wt2pc","kube-system/konnectivity-agent-929rz","openshift-insights/insights-operator-7c4986d449-jrhw4","openshift-insights/periodic-gathering-jr4s9-d8s6t","openshift-service-ca/service-ca-5fffc97bdf-mzhxr","openshift-kube-storage-version-migrator/migrator-66df5d496b-f2dgh","openshift-image-registry/image-registry-55fdbc6f6f-ngftj","kube-system/kas-connection-checker-685895c7c9-2xctf","openshift-ovn-kubernetes/ovnkube-node-vtgcr","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq","kube-system/kas-connection-checker-685895c7c9-6pjx4","openshift-cluster-node-tuning-operator/tuned-ld4hs","kube-system/kas-connection-checker-685895c7c9-k6p2v","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8","openshift-ingress-canary/ingress-canary-b6fbt","openshift-multus/multus-additional-cni-plugins-w2t6m","openshift-kube-storage-version-migrator/migrator-66df5d496b-27sld","openshift-service-ca-operator/service-ca-operator-544d749c8c-88kh6","openshift-network-diagnostics/network-check-target-6gzkm","openshift-dns/dns-default-p8hw6"] Oct 30 21:49:58.663727 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.663699 6006 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:49:58.675703 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.675671 6006 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:49:58.675859 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.675840 6006 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:49:58.686215 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.686190 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/64086350-55f6-4da5-a871-3f23c1755371-konnectivity-ca\") pod \"konnectivity-agent-929rz\" (UID: \"64086350-55f6-4da5-a871-3f23c1755371\") " pod="kube-system/konnectivity-agent-929rz" Oct 30 21:49:58.686323 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.686217 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc5dn\" (UniqueName: \"kubernetes.io/projected/13fd2c81-54ab-4afa-9afe-68094563bf08-kube-api-access-vc5dn\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.686323 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.686238 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-systemd-units\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.686323 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.686275 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8b60900b-7a0e-4039-97af-c8157ae19ca0-archives-path\") pod \"periodic-gathering-jr4s9-d8s6t\" (UID: \"8b60900b-7a0e-4039-97af-c8157ae19ca0\") " pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" Oct 30 21:49:58.686455 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.686344 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ba3d2b-2d12-458c-9984-1dd82c275e10-service-ca-bundle\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:58.686455 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.686387 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8b60900b-7a0e-4039-97af-c8157ae19ca0-archives-path\") pod \"periodic-gathering-jr4s9-d8s6t\" (UID: \"8b60900b-7a0e-4039-97af-c8157ae19ca0\") " pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" Oct 30 21:49:58.686455 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.686440 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffa64e38-3f1b-49af-913f-139addefa359-cni-binary-copy\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:49:58.686553 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.686528 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-run-netns\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.686597 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.686567 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkjjr\" (UniqueName: \"kubernetes.io/projected/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51-kube-api-access-lkjjr\") pod \"console-operator-7cd56b8dbc-8w8b5\" (UID: \"5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:49:58.686644 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.686596 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-stats-auth\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:58.686644 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.686621 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0cece9c-2008-49ce-87de-4804d4e7de08-serving-cert\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:58.686738 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.686648 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-var-lib-openvswitch\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.686738 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.686689 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-etc-openvswitch\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.686828 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.686741 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:58.686828 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.686766 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4-hosts-file\") pod \"node-resolver-74zqz\" (UID: \"1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4\") " pod="openshift-dns/node-resolver-74zqz" Oct 30 21:49:58.686828 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.686786 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0cece9c-2008-49ce-87de-4804d4e7de08-tmp\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:58.686828 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.686803 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0cece9c-2008-49ce-87de-4804d4e7de08-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:58.686971 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.686837 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3458688b-8142-492c-ab8b-bffec754c8c8-image-registry-private-configuration\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:58.686971 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.686917 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7aa1b80f-115e-4708-bc98-88d13ad66860-env-overrides\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.686971 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.686937 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0cece9c-2008-49ce-87de-4804d4e7de08-tmp\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:58.686971 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.686946 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-w7pwq\" (UID: \"fc3ff956-6466-46e3-b734-13a3ac148d36\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq" Oct 30 21:49:58.687085 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.686972 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hnzg\" (UniqueName: \"kubernetes.io/projected/d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2-kube-api-access-9hnzg\") pod \"service-ca-5fffc97bdf-mzhxr\" (UID: \"d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mzhxr" Oct 30 21:49:58.687085 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.687006 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1d430f7e-00f1-4b9e-9427-3e9971cfff31-kubelet-config\") pod \"global-pull-secret-syncer-lfq7z\" (UID: \"1d430f7e-00f1-4b9e-9427-3e9971cfff31\") " pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:49:58.687085 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.687053 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ceedf44e-88c1-49cf-901e-75558703c361-host\") pod \"node-ca-x4655\" (UID: \"ceedf44e-88c1-49cf-901e-75558703c361\") " pod="openshift-image-registry/node-ca-x4655" Oct 30 21:49:58.687218 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.687202 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-run-netns\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.687297 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.687237 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7aa1b80f-115e-4708-bc98-88d13ad66860-ovnkube-config\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.687297 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.687286 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/84bfdb37-6346-4384-8d97-d43aba6f03fe-installation-pull-secrets\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:58.687457 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.687348 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3458688b-8142-492c-ab8b-bffec754c8c8-bound-sa-token\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:58.687457 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.687366 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/35a08208-7245-4e1b-9c0b-9e825a808365-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-55pjz\" (UID: \"35a08208-7245-4e1b-9c0b-9e825a808365\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-55pjz" Oct 30 21:49:58.687457 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.687404 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-kubernetes\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.687457 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.687419 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-slash\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.687457 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.687447 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3458688b-8142-492c-ab8b-bffec754c8c8-registry-certificates\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:58.687681 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.687488 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-multus-socket-dir-parent\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.687681 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.687543 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-run-k8s-cni-cncf-io\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.687681 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.687577 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2-signing-key\") pod \"service-ca-5fffc97bdf-mzhxr\" (UID: \"d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mzhxr" Oct 30 21:49:58.687681 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.687613 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58wrk\" (UniqueName: \"kubernetes.io/projected/11ba3d2b-2d12-458c-9984-1dd82c275e10-kube-api-access-58wrk\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:58.687681 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.687670 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-run-ovn\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.687921 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.687700 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-node-log\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.687921 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.687723 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/95097323-8907-45bc-8967-0de718adefbd-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-hmttl\" (UID: \"95097323-8907-45bc-8967-0de718adefbd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl" Oct 30 21:49:58.687921 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.687828 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-device-dir\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:49:58.687921 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.687855 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7fe5efb-7029-41b8-8bbe-f37f49410f27-cert\") pod \"ingress-canary-b6fbt\" (UID: \"c7fe5efb-7029-41b8-8bbe-f37f49410f27\") " pod="openshift-ingress-canary/ingress-canary-b6fbt" Oct 30 21:49:58.687921 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.687893 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn7gh\" (UniqueName: \"kubernetes.io/projected/c7fe5efb-7029-41b8-8bbe-f37f49410f27-kube-api-access-fn7gh\") pod \"ingress-canary-b6fbt\" (UID: \"c7fe5efb-7029-41b8-8bbe-f37f49410f27\") " pod="openshift-ingress-canary/ingress-canary-b6fbt" Oct 30 21:49:58.688163 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.687923 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-sysctl-d\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.688163 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.687947 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-hostroot\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.688163 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.687983 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7zd\" (UniqueName: \"kubernetes.io/projected/0dce5b38-e1a7-41e3-8100-96af4e3c2b06-kube-api-access-jr7zd\") pod \"kas-connection-checker-685895c7c9-6pjx4\" (UID: \"0dce5b38-e1a7-41e3-8100-96af4e3c2b06\") " pod="kube-system/kas-connection-checker-685895c7c9-6pjx4" Oct 30 21:49:58.688163 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688008 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9lcg\" (UniqueName: \"kubernetes.io/projected/fc3ff956-6466-46e3-b734-13a3ac148d36-kube-api-access-k9lcg\") pod \"cluster-samples-operator-6ddb6c8b94-w7pwq\" (UID: \"fc3ff956-6466-46e3-b734-13a3ac148d36\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq" Oct 30 21:49:58.688163 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688025 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-etc-selinux\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:49:58.688163 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688062 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ffa64e38-3f1b-49af-913f-139addefa359-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:49:58.688163 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688121 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3458688b-8142-492c-ab8b-bffec754c8c8-ca-trust-extracted\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:58.688163 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688152 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5gt6\" (UniqueName: \"kubernetes.io/projected/ce930258-60d9-4988-913c-2f3cb78eac8d-kube-api-access-v5gt6\") pod \"downloads-6dd789c8b7-tqt4b\" (UID: \"ce930258-60d9-4988-913c-2f3cb78eac8d\") " pod="openshift-console/downloads-6dd789c8b7-tqt4b" Oct 30 21:49:58.688562 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688188 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13fd2c81-54ab-4afa-9afe-68094563bf08-cni-binary-copy\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.688562 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688206 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-var-lib-cni-bin\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.688562 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688210 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3458688b-8142-492c-ab8b-bffec754c8c8-ca-trust-extracted\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:58.688562 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688236 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvxrm\" (UniqueName: \"kubernetes.io/projected/7aa1b80f-115e-4708-bc98-88d13ad66860-kube-api-access-qvxrm\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.688562 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688278 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8ctq\" (UniqueName: \"kubernetes.io/projected/4579f026-39bd-465c-b0db-41676370ba55-kube-api-access-l8ctq\") pod \"volume-data-source-validator-58c9f5f46d-p54hn\" (UID: \"4579f026-39bd-465c-b0db-41676370ba55\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p54hn" Oct 30 21:49:58.688562 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688321 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4-tmp-dir\") pod \"node-resolver-74zqz\" (UID: \"1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4\") " pod="openshift-dns/node-resolver-74zqz" Oct 30 21:49:58.688562 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688440 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vss2v\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-kube-api-access-vss2v\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:58.688562 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688452 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4-tmp-dir\") pod \"node-resolver-74zqz\" (UID: \"1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4\") " pod="openshift-dns/node-resolver-74zqz" Oct 30 21:49:58.688562 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688474 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-run\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.688562 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688498 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-lib-modules\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.688562 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688544 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-run-openvswitch\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.689048 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688573 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-data-volume\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:58.689048 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688609 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgd6c\" (UniqueName: \"kubernetes.io/projected/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-kube-api-access-hgd6c\") pod \"dns-default-p8hw6\" (UID: \"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3\") " pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:58.689048 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688628 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51-serving-cert\") pod \"console-operator-7cd56b8dbc-8w8b5\" (UID: \"5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:49:58.689048 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688650 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/84bfdb37-6346-4384-8d97-d43aba6f03fe-ca-trust-extracted\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:58.689048 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688676 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/84bfdb37-6346-4384-8d97-d43aba6f03fe-trusted-ca\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:58.689048 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688683 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-data-volume\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:58.689048 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688703 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcm8z\" (UniqueName: \"kubernetes.io/projected/8b60900b-7a0e-4039-97af-c8157ae19ca0-kube-api-access-fcm8z\") pod \"periodic-gathering-jr4s9-d8s6t\" (UID: \"8b60900b-7a0e-4039-97af-c8157ae19ca0\") " pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" Oct 30 21:49:58.689048 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688729 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ffa64e38-3f1b-49af-913f-139addefa359-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:49:58.689048 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688757 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9161cdbd-c44f-4a8c-b918-107fc7001f3a-host-slash\") pod \"iptables-alerter-gft82\" (UID: \"9161cdbd-c44f-4a8c-b918-107fc7001f3a\") " pod="openshift-network-operator/iptables-alerter-gft82" Oct 30 21:49:58.689048 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688761 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/84bfdb37-6346-4384-8d97-d43aba6f03fe-ca-trust-extracted\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:58.689048 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688808 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk7l4\" (UniqueName: \"kubernetes.io/projected/550e690a-605e-47a4-ba95-74bcd26e0cc3-kube-api-access-kk7l4\") pod \"network-check-target-6gzkm\" (UID: \"550e690a-605e-47a4-ba95-74bcd26e0cc3\") " pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:49:58.689048 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688933 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-run-systemd\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.689048 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.688969 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-config-volume\") pod \"dns-default-p8hw6\" (UID: \"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3\") " pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:58.689048 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689007 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j58ln\" (UniqueName: \"kubernetes.io/projected/95097323-8907-45bc-8967-0de718adefbd-kube-api-access-j58ln\") pod \"cluster-monitoring-operator-884576c57-hmttl\" (UID: \"95097323-8907-45bc-8967-0de718adefbd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl" Oct 30 21:49:58.689048 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689053 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c0cece9c-2008-49ce-87de-4804d4e7de08-snapshots\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689082 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0cece9c-2008-49ce-87de-4804d4e7de08-service-ca-bundle\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689116 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-tuned\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689138 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689145 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/84bfdb37-6346-4384-8d97-d43aba6f03fe-image-registry-private-configuration\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689171 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1d430f7e-00f1-4b9e-9427-3e9971cfff31-dbus\") pod \"global-pull-secret-syncer-lfq7z\" (UID: \"1d430f7e-00f1-4b9e-9427-3e9971cfff31\") " pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689183 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c0cece9c-2008-49ce-87de-4804d4e7de08-snapshots\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689195 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-sys\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689240 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-tuned\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689277 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-run-multus-certs\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689328 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-kubelet\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689353 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689365 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689375 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689396 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689412 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2-signing-cabundle\") pod \"service-ca-5fffc97bdf-mzhxr\" (UID: \"d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mzhxr" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689430 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689488 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689491 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689506 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-4wbzb\"" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689508 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ntfhf\"" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689482 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9161cdbd-c44f-4a8c-b918-107fc7001f3a-iptables-alerter-script\") pod \"iptables-alerter-gft82\" (UID: \"9161cdbd-c44f-4a8c-b918-107fc7001f3a\") " pod="openshift-network-operator/iptables-alerter-gft82" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689552 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-var-lib-kubelet\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689592 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-826wq\" (UniqueName: \"kubernetes.io/projected/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-kube-api-access-826wq\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689629 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgccp\" (UniqueName: \"kubernetes.io/projected/11109f61-f4f0-4253-aacb-886c353a5772-kube-api-access-vgccp\") pod \"service-ca-operator-544d749c8c-88kh6\" (UID: \"11109f61-f4f0-4253-aacb-886c353a5772\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88kh6" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689649 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-sys-fs\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689666 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fgkx\" (UniqueName: \"kubernetes.io/projected/ffa64e38-3f1b-49af-913f-139addefa359-kube-api-access-5fgkx\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:49:58.689775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689682 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-systemd\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.690950 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689809 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7aa1b80f-115e-4708-bc98-88d13ad66860-ovnkube-script-lib\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.690950 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689826 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:49:58.690950 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689864 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2b29\" (UniqueName: \"kubernetes.io/projected/8bf0fd8f-2294-4729-9f49-a151de8627aa-kube-api-access-r2b29\") pod \"kas-connection-checker-685895c7c9-2xctf\" (UID: \"8bf0fd8f-2294-4729-9f49-a151de8627aa\") " pod="kube-system/kas-connection-checker-685895c7c9-2xctf" Oct 30 21:49:58.690950 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689882 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:58.690950 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689925 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-sysctl-conf\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.690950 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689951 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a32b60f2-3000-4fe8-a928-955de8c76c82-tmp\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.690950 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.689997 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95097323-8907-45bc-8967-0de718adefbd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hmttl\" (UID: \"95097323-8907-45bc-8967-0de718adefbd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl" Oct 30 21:49:58.690950 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.690027 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a32b60f2-3000-4fe8-a928-955de8c76c82-tmp\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.690950 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.690037 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-b95t7\"" Oct 30 21:49:58.690950 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.690028 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-certificates\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:58.690950 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.690086 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/35a08208-7245-4e1b-9c0b-9e825a808365-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-55pjz\" (UID: \"35a08208-7245-4e1b-9c0b-9e825a808365\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-55pjz" Oct 30 21:49:58.690950 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.690113 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-socket-dir\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:49:58.690950 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.690126 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:49:58.690950 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.690141 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-bound-sa-token\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:58.690950 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.690169 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxnj9\" (UniqueName: \"kubernetes.io/projected/3458688b-8142-492c-ab8b-bffec754c8c8-kube-api-access-hxnj9\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:58.690950 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.690283 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95097323-8907-45bc-8967-0de718adefbd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hmttl\" (UID: \"95097323-8907-45bc-8967-0de718adefbd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl" Oct 30 21:49:58.690950 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.690729 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-run-ovn-kubernetes\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.690950 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.690769 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-default-certificate\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:58.690950 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.690849 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8df7e7fd-be92-4d37-af33-930b7b36cbe6-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-bc452\" (UID: \"8df7e7fd-be92-4d37-af33-930b7b36cbe6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bc452" Oct 30 21:49:58.691783 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.690978 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-sysconfig\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.691783 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.691015 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6k4l\" (UniqueName: \"kubernetes.io/projected/a32b60f2-3000-4fe8-a928-955de8c76c82-kube-api-access-r6k4l\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.691783 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.691040 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-var-lib-kubelet\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.691783 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.691212 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8258\" (UniqueName: \"kubernetes.io/projected/35a08208-7245-4e1b-9c0b-9e825a808365-kube-api-access-h8258\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-55pjz\" (UID: \"35a08208-7245-4e1b-9c0b-9e825a808365\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-55pjz" Oct 30 21:49:58.691783 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.691271 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-multus-cni-dir\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.691783 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.691304 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-log-socket\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.691783 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.691365 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-cni-netd\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.691783 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.691511 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7aa1b80f-115e-4708-bc98-88d13ad66860-ovn-node-metrics-cert\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.691783 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.691547 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-crio-socket\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:58.691783 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.691590 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffa64e38-3f1b-49af-913f-139addefa359-cnibin\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:49:58.691783 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.691722 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51-config\") pod \"console-operator-7cd56b8dbc-8w8b5\" (UID: \"5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:49:58.691783 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.691762 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67x9l\" (UniqueName: \"kubernetes.io/projected/226c41d5-de84-4950-978f-a1f0b4cbc319-kube-api-access-67x9l\") pod \"network-check-source-6f78d59f99-989z4\" (UID: \"226c41d5-de84-4950-978f-a1f0b4cbc319\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-989z4" Oct 30 21:49:58.692393 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.691796 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3458688b-8142-492c-ab8b-bffec754c8c8-registry-tls\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:58.692393 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.691889 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/47917159-29d7-45b9-bbf2-13cef4408de9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5xd2z\" (UID: \"47917159-29d7-45b9-bbf2-13cef4408de9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xd2z" Oct 30 21:49:58.692393 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.691936 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-metrics-certs\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:58.692393 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.691991 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/64086350-55f6-4da5-a871-3f23c1755371-agent-certs\") pod \"konnectivity-agent-929rz\" (UID: \"64086350-55f6-4da5-a871-3f23c1755371\") " pod="kube-system/konnectivity-agent-929rz" Oct 30 21:49:58.692393 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.692035 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thhdq\" (UniqueName: \"kubernetes.io/projected/9df43c88-a01f-4ec7-a385-d5afcf6f9e6c-kube-api-access-thhdq\") pod \"kas-connection-checker-685895c7c9-k6p2v\" (UID: \"9df43c88-a01f-4ec7-a385-d5afcf6f9e6c\") " pod="kube-system/kas-connection-checker-685895c7c9-k6p2v" Oct 30 21:49:58.692393 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.692175 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ceedf44e-88c1-49cf-901e-75558703c361-serviceca\") pod \"node-ca-x4655\" (UID: \"ceedf44e-88c1-49cf-901e-75558703c361\") " pod="openshift-image-registry/node-ca-x4655" Oct 30 21:49:58.692393 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.692205 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/47917159-29d7-45b9-bbf2-13cef4408de9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5xd2z\" (UID: \"47917159-29d7-45b9-bbf2-13cef4408de9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xd2z" Oct 30 21:49:58.692393 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.692240 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11109f61-f4f0-4253-aacb-886c353a5772-serving-cert\") pod \"service-ca-operator-544d749c8c-88kh6\" (UID: \"11109f61-f4f0-4253-aacb-886c353a5772\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88kh6" Oct 30 21:49:58.693007 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.692979 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-metrics-tls\") pod \"dns-default-p8hw6\" (UID: \"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3\") " pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:58.693137 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.693036 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret\") pod \"global-pull-secret-syncer-lfq7z\" (UID: \"1d430f7e-00f1-4b9e-9427-3e9971cfff31\") " pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:49:58.693137 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.693067 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce930258-60d9-4988-913c-2f3cb78eac8d-tmp\") pod \"downloads-6dd789c8b7-tqt4b\" (UID: \"ce930258-60d9-4988-913c-2f3cb78eac8d\") " pod="openshift-console/downloads-6dd789c8b7-tqt4b" Oct 30 21:49:58.693272 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.693205 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-var-lib-cni-multus\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.693272 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.693229 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce930258-60d9-4988-913c-2f3cb78eac8d-tmp\") pod \"downloads-6dd789c8b7-tqt4b\" (UID: \"ce930258-60d9-4988-913c-2f3cb78eac8d\") " pod="openshift-console/downloads-6dd789c8b7-tqt4b" Oct 30 21:49:58.693385 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.693264 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:58.693385 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.693306 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-modprobe-d\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.693385 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.693341 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-system-cni-dir\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.693530 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.693382 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs\") pod \"network-metrics-daemon-69j9v\" (UID: \"bd3b421a-9313-4f15-b81f-900717685fd6\") " pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:49:58.693530 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.693421 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdhdv\" (UniqueName: \"kubernetes.io/projected/f71e935d-f091-4f16-820b-8d7fc632f003-kube-api-access-vdhdv\") pod \"migrator-66df5d496b-f2dgh\" (UID: \"f71e935d-f091-4f16-820b-8d7fc632f003\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f2dgh" Oct 30 21:49:58.693530 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.693457 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f54bb\" (UniqueName: \"kubernetes.io/projected/6e73e56e-9619-4b97-bf20-a6291acf6355-kube-api-access-f54bb\") pod \"migrator-66df5d496b-27sld\" (UID: \"6e73e56e-9619-4b97-bf20-a6291acf6355\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-27sld" Oct 30 21:49:58.693530 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.693491 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3458688b-8142-492c-ab8b-bffec754c8c8-trusted-ca\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:58.693709 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.693549 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-tmp-dir\") pod \"dns-default-p8hw6\" (UID: \"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3\") " pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:58.693855 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.693782 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q7c2\" (UniqueName: \"kubernetes.io/projected/1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4-kube-api-access-7q7c2\") pod \"node-resolver-74zqz\" (UID: \"1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4\") " pod="openshift-dns/node-resolver-74zqz" Oct 30 21:49:58.693915 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.693881 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-registration-dir\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:49:58.693976 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.693916 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-tmp-dir\") pod \"dns-default-p8hw6\" (UID: \"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3\") " pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:58.693976 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.693924 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hz67\" (UniqueName: \"kubernetes.io/projected/9161cdbd-c44f-4a8c-b918-107fc7001f3a-kube-api-access-9hz67\") pod \"iptables-alerter-gft82\" (UID: \"9161cdbd-c44f-4a8c-b918-107fc7001f3a\") " pod="openshift-network-operator/iptables-alerter-gft82" Oct 30 21:49:58.693976 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.693957 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-etc-kubernetes\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.694107 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.693991 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/13fd2c81-54ab-4afa-9afe-68094563bf08-multus-daemon-config\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.694107 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.694069 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-cni-bin\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.694107 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.694103 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b60900b-7a0e-4039-97af-c8157ae19ca0-service-ca-bundle\") pod \"periodic-gathering-jr4s9-d8s6t\" (UID: \"8b60900b-7a0e-4039-97af-c8157ae19ca0\") " pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" Oct 30 21:49:58.694237 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.694125 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 21:49:58.694237 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.694140 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3458688b-8142-492c-ab8b-bffec754c8c8-installation-pull-secrets\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:58.694237 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.694173 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2qx6\" (UniqueName: \"kubernetes.io/projected/bd3b421a-9313-4f15-b81f-900717685fd6-kube-api-access-j2qx6\") pod \"network-metrics-daemon-69j9v\" (UID: \"bd3b421a-9313-4f15-b81f-900717685fd6\") " pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:49:58.694237 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.694231 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11109f61-f4f0-4253-aacb-886c353a5772-config\") pod \"service-ca-operator-544d749c8c-88kh6\" (UID: \"11109f61-f4f0-4253-aacb-886c353a5772\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88kh6" Oct 30 21:49:58.694431 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.694313 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:49:58.694431 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.694350 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j92gp\" (UniqueName: \"kubernetes.io/projected/c0cece9c-2008-49ce-87de-4804d4e7de08-kube-api-access-j92gp\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:58.694596 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.694574 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffa64e38-3f1b-49af-913f-139addefa359-system-cni-dir\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:49:58.694652 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.694621 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffa64e38-3f1b-49af-913f-139addefa359-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:49:58.694701 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.694660 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8pl2\" (UniqueName: \"kubernetes.io/projected/ceedf44e-88c1-49cf-901e-75558703c361-kube-api-access-s8pl2\") pod \"node-ca-x4655\" (UID: \"ceedf44e-88c1-49cf-901e-75558703c361\") " pod="openshift-image-registry/node-ca-x4655" Oct 30 21:49:58.694701 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.694693 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b60900b-7a0e-4039-97af-c8157ae19ca0-trusted-ca-bundle\") pod \"periodic-gathering-jr4s9-d8s6t\" (UID: \"8b60900b-7a0e-4039-97af-c8157ae19ca0\") " pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" Oct 30 21:49:58.694799 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.694731 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51-trusted-ca\") pod \"console-operator-7cd56b8dbc-8w8b5\" (UID: \"5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:49:58.694799 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.694782 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-w7pwq\" (UID: \"fc3ff956-6466-46e3-b734-13a3ac148d36\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq" Oct 30 21:49:58.694963 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.694927 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw4xv\" (UniqueName: \"kubernetes.io/projected/b2f2c486-d661-4d59-981c-0b95a1f66fb4-kube-api-access-sw4xv\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:49:58.695064 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.695012 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-w7pwq\" (UID: \"fc3ff956-6466-46e3-b734-13a3ac148d36\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq" Oct 30 21:49:58.695120 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.695095 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-host\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.695894 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.695265 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-cnibin\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.695894 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.695319 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-multus-conf-dir\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.695894 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.695502 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:49:58.697183 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.697162 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fc3ff956-6466-46e3-b734-13a3ac148d36-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-w7pwq\" (UID: \"fc3ff956-6466-46e3-b734-13a3ac148d36\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq" Oct 30 21:49:58.697276 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.697179 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0cece9c-2008-49ce-87de-4804d4e7de08-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:58.698134 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.698117 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/95097323-8907-45bc-8967-0de718adefbd-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-hmttl\" (UID: \"95097323-8907-45bc-8967-0de718adefbd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl" Oct 30 21:49:58.705840 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.705823 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b60900b-7a0e-4039-97af-c8157ae19ca0-trusted-ca-bundle\") pod \"periodic-gathering-jr4s9-d8s6t\" (UID: \"8b60900b-7a0e-4039-97af-c8157ae19ca0\") " pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" Oct 30 21:49:58.728919 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.728891 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 21:49:58.729167 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.729149 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 21:49:58.729337 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.729322 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-zxjct\"" Oct 30 21:49:58.729431 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.729415 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 21:49:58.730528 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.730509 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/35a08208-7245-4e1b-9c0b-9e825a808365-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-55pjz\" (UID: \"35a08208-7245-4e1b-9c0b-9e825a808365\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-55pjz" Oct 30 21:49:58.731153 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.731134 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-default-certificate\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:58.731399 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.731385 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:49:58.732757 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.732739 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/64086350-55f6-4da5-a871-3f23c1755371-agent-certs\") pod \"konnectivity-agent-929rz\" (UID: \"64086350-55f6-4da5-a871-3f23c1755371\") " pod="kube-system/konnectivity-agent-929rz" Oct 30 21:49:58.734308 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.734290 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-92jf8\"" Oct 30 21:49:58.736920 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.736900 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-stats-auth\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:58.737056 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.737040 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 21:49:58.737120 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.737088 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 21:49:58.737173 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.737131 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jbflf\"" Oct 30 21:49:58.737283 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.737259 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:49:58.738554 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.738539 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/35a08208-7245-4e1b-9c0b-9e825a808365-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-55pjz\" (UID: \"35a08208-7245-4e1b-9c0b-9e825a808365\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-55pjz" Oct 30 21:49:58.739925 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.739907 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0cece9c-2008-49ce-87de-4804d4e7de08-service-ca-bundle\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:58.741117 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.741101 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-x2j52\"" Oct 30 21:49:58.745858 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.745837 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b60900b-7a0e-4039-97af-c8157ae19ca0-service-ca-bundle\") pod \"periodic-gathering-jr4s9-d8s6t\" (UID: \"8b60900b-7a0e-4039-97af-c8157ae19ca0\") " pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" Oct 30 21:49:58.746591 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.746564 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 21:49:58.746674 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.746598 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7d2ft\"" Oct 30 21:49:58.750293 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.750266 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cmhxq\"" Oct 30 21:49:58.750379 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.750303 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dvh7z\"" Oct 30 21:49:58.751392 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.751372 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9mttd\"" Oct 30 21:49:58.751853 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.751770 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 21:49:58.751853 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.751774 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 21:49:58.751853 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.751817 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:49:58.751982 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.751854 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 21:49:58.753145 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.753126 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/47917159-29d7-45b9-bbf2-13cef4408de9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5xd2z\" (UID: \"47917159-29d7-45b9-bbf2-13cef4408de9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xd2z" Oct 30 21:49:58.753216 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.753172 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51-config\") pod \"console-operator-7cd56b8dbc-8w8b5\" (UID: \"5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:49:58.753980 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.753967 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 21:49:58.754084 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.754074 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:49:58.754126 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.754082 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:49:58.754126 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.754101 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:49:58.754265 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.754233 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:49:58.754314 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.754236 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:49:58.754468 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.754434 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:49:58.754540 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.754488 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:49:58.754540 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.754495 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:49:58.754614 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.754580 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:49:58.754614 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.754587 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/13fd2c81-54ab-4afa-9afe-68094563bf08-multus-daemon-config\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.754614 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.754600 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:49:58.754614 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.754598 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:49:58.754773 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.754621 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:49:58.754773 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.754622 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:49:58.754773 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.754696 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:49:58.754888 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.754813 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:49:58.754888 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.754871 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:49:58.754957 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.754894 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:49:58.754957 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.754928 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:49:58.754957 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.754942 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:49:58.755475 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.755387 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 21:49:58.755475 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.755388 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:49:58.755475 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.755393 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:49:58.755475 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.755429 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 21:49:58.755475 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.755431 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 21:49:58.755475 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.755433 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 21:49:58.755475 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.755473 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 21:49:58.755759 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.755508 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 21:49:58.755759 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.755555 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 21:49:58.755759 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.755584 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:49:58.755759 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.755608 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 21:49:58.755759 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.755636 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:49:58.755759 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.755659 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:49:58.755759 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.755674 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:49:58.756054 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.755784 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 21:49:58.756054 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.755804 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:49:58.756054 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.755834 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 21:49:58.756054 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.755850 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:49:58.756054 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.755944 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 21:49:58.756283 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.756175 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:49:58.756283 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.756235 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:49:58.756359 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.756282 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:49:58.756397 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.756359 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:49:58.756657 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.756635 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffa64e38-3f1b-49af-913f-139addefa359-cni-binary-copy\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:49:58.756730 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.756670 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ba3d2b-2d12-458c-9984-1dd82c275e10-service-ca-bundle\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:58.756987 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.756971 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:49:58.757026 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.756977 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:49:58.757026 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757016 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:49:58.757026 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757022 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:49:58.757121 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757038 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3458688b-8142-492c-ab8b-bffec754c8c8-image-registry-private-configuration\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:58.757277 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757260 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:49:58.757328 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757316 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:49:58.757375 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757344 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-7fn9v\"" Oct 30 21:49:58.757375 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757349 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-nrl58\"" Oct 30 21:49:58.757375 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757366 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 21:49:58.757512 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757371 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 21:49:58.757512 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757384 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-zp2pt\"" Oct 30 21:49:58.757512 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757423 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-w6rhz\"" Oct 30 21:49:58.757512 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757453 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-sjphq\"" Oct 30 21:49:58.757512 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757455 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 21:49:58.757512 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757473 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:49:58.757512 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757477 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gbc4r\"" Oct 30 21:49:58.757512 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757499 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6gwn9\"" Oct 30 21:49:58.757858 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757522 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9bjlv\"" Oct 30 21:49:58.757858 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757526 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:49:58.757858 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757546 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-d8p2x\"" Oct 30 21:49:58.757858 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757550 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 21:49:58.757858 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757564 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9mgc8\"" Oct 30 21:49:58.757858 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757566 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7aa1b80f-115e-4708-bc98-88d13ad66860-ovnkube-config\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.757858 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757602 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 21:49:58.757858 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757501 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:49:58.757858 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757663 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-qzl5d\"" Oct 30 21:49:58.757858 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757775 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3458688b-8142-492c-ab8b-bffec754c8c8-registry-certificates\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:58.757858 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757783 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8rvzh\"" Oct 30 21:49:58.757858 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757808 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:49:58.757858 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757841 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2-signing-key\") pod \"service-ca-5fffc97bdf-mzhxr\" (UID: \"d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mzhxr" Oct 30 21:49:58.758402 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757885 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 21:49:58.758402 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757931 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nwhdt\"" Oct 30 21:49:58.758402 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757939 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:49:58.758402 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.757991 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hm4gw\"" Oct 30 21:49:58.758402 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.758106 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-52nbt\"" Oct 30 21:49:58.758402 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.758136 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:49:58.758402 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.758173 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7fe5efb-7029-41b8-8bbe-f37f49410f27-cert\") pod \"ingress-canary-b6fbt\" (UID: \"c7fe5efb-7029-41b8-8bbe-f37f49410f27\") " pod="openshift-ingress-canary/ingress-canary-b6fbt" Oct 30 21:49:58.758402 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.758232 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-pf5gv\"" Oct 30 21:49:58.758402 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.758241 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8lg5r\"" Oct 30 21:49:58.758402 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.758271 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bbh9g\"" Oct 30 21:49:58.758402 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.758361 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ffa64e38-3f1b-49af-913f-139addefa359-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:49:58.758402 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.758401 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:49:58.758736 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.758411 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:49:58.758736 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.758412 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:49:58.758736 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.758428 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:49:58.758736 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.758408 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13fd2c81-54ab-4afa-9afe-68094563bf08-cni-binary-copy\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.759486 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.759471 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/84bfdb37-6346-4384-8d97-d43aba6f03fe-image-registry-private-configuration\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:58.759553 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.759505 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-config-volume\") pod \"dns-default-p8hw6\" (UID: \"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3\") " pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:58.759603 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.759552 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ffa64e38-3f1b-49af-913f-139addefa359-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:49:58.759655 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.759614 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51-serving-cert\") pod \"console-operator-7cd56b8dbc-8w8b5\" (UID: \"5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:49:58.759740 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.759726 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2-signing-cabundle\") pod \"service-ca-5fffc97bdf-mzhxr\" (UID: \"d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mzhxr" Oct 30 21:49:58.759780 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.759722 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9161cdbd-c44f-4a8c-b918-107fc7001f3a-iptables-alerter-script\") pod \"iptables-alerter-gft82\" (UID: \"9161cdbd-c44f-4a8c-b918-107fc7001f3a\") " pod="openshift-network-operator/iptables-alerter-gft82" Oct 30 21:49:58.760763 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.760745 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7aa1b80f-115e-4708-bc98-88d13ad66860-ovnkube-script-lib\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.760816 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.760762 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:58.760816 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.760797 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-certificates\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:58.761895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.761877 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8df7e7fd-be92-4d37-af33-930b7b36cbe6-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-bc452\" (UID: \"8df7e7fd-be92-4d37-af33-930b7b36cbe6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bc452" Oct 30 21:49:58.763091 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.763068 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/47917159-29d7-45b9-bbf2-13cef4408de9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5xd2z\" (UID: \"47917159-29d7-45b9-bbf2-13cef4408de9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xd2z" Oct 30 21:49:58.763164 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.763091 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11ba3d2b-2d12-458c-9984-1dd82c275e10-metrics-certs\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:58.763164 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.763076 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ceedf44e-88c1-49cf-901e-75558703c361-serviceca\") pod \"node-ca-x4655\" (UID: \"ceedf44e-88c1-49cf-901e-75558703c361\") " pod="openshift-image-registry/node-ca-x4655" Oct 30 21:49:58.763164 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.763116 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7aa1b80f-115e-4708-bc98-88d13ad66860-ovn-node-metrics-cert\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.763164 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.763120 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3458688b-8142-492c-ab8b-bffec754c8c8-registry-tls\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:58.763326 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.763191 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11109f61-f4f0-4253-aacb-886c353a5772-serving-cert\") pod \"service-ca-operator-544d749c8c-88kh6\" (UID: \"11109f61-f4f0-4253-aacb-886c353a5772\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88kh6" Oct 30 21:49:58.763363 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.763352 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-metrics-tls\") pod \"dns-default-p8hw6\" (UID: \"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3\") " pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:58.763472 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.763460 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d430f7e-00f1-4b9e-9427-3e9971cfff31-original-pull-secret\") pod \"global-pull-secret-syncer-lfq7z\" (UID: \"1d430f7e-00f1-4b9e-9427-3e9971cfff31\") " pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:49:58.763572 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.763558 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:58.764357 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.764339 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd3b421a-9313-4f15-b81f-900717685fd6-metrics-certs\") pod \"network-metrics-daemon-69j9v\" (UID: \"bd3b421a-9313-4f15-b81f-900717685fd6\") " pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:49:58.765419 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.765400 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3458688b-8142-492c-ab8b-bffec754c8c8-installation-pull-secrets\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:58.765419 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.765414 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11109f61-f4f0-4253-aacb-886c353a5772-config\") pod \"service-ca-operator-544d749c8c-88kh6\" (UID: \"11109f61-f4f0-4253-aacb-886c353a5772\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88kh6" Oct 30 21:49:58.766034 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.766015 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51-trusted-ca\") pod \"console-operator-7cd56b8dbc-8w8b5\" (UID: \"5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:49:58.766561 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.766546 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/64086350-55f6-4da5-a871-3f23c1755371-konnectivity-ca\") pod \"konnectivity-agent-929rz\" (UID: \"64086350-55f6-4da5-a871-3f23c1755371\") " pod="kube-system/konnectivity-agent-929rz" Oct 30 21:49:58.767712 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.767695 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/84bfdb37-6346-4384-8d97-d43aba6f03fe-installation-pull-secrets\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:58.767764 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.767710 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:58.767764 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.767723 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7aa1b80f-115e-4708-bc98-88d13ad66860-env-overrides\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.767955 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.767938 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0cece9c-2008-49ce-87de-4804d4e7de08-serving-cert\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:58.776425 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.776407 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vss2v\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-kube-api-access-vss2v\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:58.777375 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.777360 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgd6c\" (UniqueName: \"kubernetes.io/projected/3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3-kube-api-access-hgd6c\") pod \"dns-default-p8hw6\" (UID: \"3e9cf4d3-e8aa-4a4b-aea3-88f4d2ab87c3\") " pod="openshift-dns/dns-default-p8hw6" Oct 30 21:49:58.778287 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.778272 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:49:58.778518 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.778480 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-tzzl2\"" Oct 30 21:49:58.778600 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.778515 6006 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:49:58.778664 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:49:58.778620 6006 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:49:58.779074 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.779056 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkjjr\" (UniqueName: \"kubernetes.io/projected/5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51-kube-api-access-lkjjr\") pod \"console-operator-7cd56b8dbc-8w8b5\" (UID: \"5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:49:58.779168 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.779149 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr7zd\" (UniqueName: \"kubernetes.io/projected/0dce5b38-e1a7-41e3-8100-96af4e3c2b06-kube-api-access-jr7zd\") pod \"kas-connection-checker-685895c7c9-6pjx4\" (UID: \"0dce5b38-e1a7-41e3-8100-96af4e3c2b06\") " pod="kube-system/kas-connection-checker-685895c7c9-6pjx4" Oct 30 21:49:58.779209 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.779194 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/84bfdb37-6346-4384-8d97-d43aba6f03fe-trusted-ca\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:58.779357 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.779343 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9lcg\" (UniqueName: \"kubernetes.io/projected/fc3ff956-6466-46e3-b734-13a3ac148d36-kube-api-access-k9lcg\") pod \"cluster-samples-operator-6ddb6c8b94-w7pwq\" (UID: \"fc3ff956-6466-46e3-b734-13a3ac148d36\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-w7pwq" Oct 30 21:49:58.780244 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.780222 6006 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:49:58.780475 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:49:58.780286 6006 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-7.us-west-2.compute.internal" Oct 30 21:49:58.780708 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.780686 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvxrm\" (UniqueName: \"kubernetes.io/projected/7aa1b80f-115e-4708-bc98-88d13ad66860-kube-api-access-qvxrm\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.780802 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.780688 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2b29\" (UniqueName: \"kubernetes.io/projected/8bf0fd8f-2294-4729-9f49-a151de8627aa-kube-api-access-r2b29\") pod \"kas-connection-checker-685895c7c9-2xctf\" (UID: \"8bf0fd8f-2294-4729-9f49-a151de8627aa\") " pod="kube-system/kas-connection-checker-685895c7c9-2xctf" Oct 30 21:49:58.780802 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.780692 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn7gh\" (UniqueName: \"kubernetes.io/projected/c7fe5efb-7029-41b8-8bbe-f37f49410f27-kube-api-access-fn7gh\") pod \"ingress-canary-b6fbt\" (UID: \"c7fe5efb-7029-41b8-8bbe-f37f49410f27\") " pod="openshift-ingress-canary/ingress-canary-b6fbt" Oct 30 21:49:58.780969 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.780940 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5gt6\" (UniqueName: \"kubernetes.io/projected/ce930258-60d9-4988-913c-2f3cb78eac8d-kube-api-access-v5gt6\") pod \"downloads-6dd789c8b7-tqt4b\" (UID: \"ce930258-60d9-4988-913c-2f3cb78eac8d\") " pod="openshift-console/downloads-6dd789c8b7-tqt4b" Oct 30 21:49:58.780969 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.780957 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58wrk\" (UniqueName: \"kubernetes.io/projected/11ba3d2b-2d12-458c-9984-1dd82c275e10-kube-api-access-58wrk\") pod \"router-default-76684dbbf-8mwh7\" (UID: \"11ba3d2b-2d12-458c-9984-1dd82c275e10\") " pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:58.781071 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.781029 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8ctq\" (UniqueName: \"kubernetes.io/projected/4579f026-39bd-465c-b0db-41676370ba55-kube-api-access-l8ctq\") pod \"volume-data-source-validator-58c9f5f46d-p54hn\" (UID: \"4579f026-39bd-465c-b0db-41676370ba55\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p54hn" Oct 30 21:49:58.781152 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.781128 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fgkx\" (UniqueName: \"kubernetes.io/projected/ffa64e38-3f1b-49af-913f-139addefa359-kube-api-access-5fgkx\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:49:58.781387 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.781369 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgccp\" (UniqueName: \"kubernetes.io/projected/11109f61-f4f0-4253-aacb-886c353a5772-kube-api-access-vgccp\") pod \"service-ca-operator-544d749c8c-88kh6\" (UID: \"11109f61-f4f0-4253-aacb-886c353a5772\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88kh6" Oct 30 21:49:58.781387 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.781375 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk7l4\" (UniqueName: \"kubernetes.io/projected/550e690a-605e-47a4-ba95-74bcd26e0cc3-kube-api-access-kk7l4\") pod \"network-check-target-6gzkm\" (UID: \"550e690a-605e-47a4-ba95-74bcd26e0cc3\") " pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:49:58.781527 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.781426 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3458688b-8142-492c-ab8b-bffec754c8c8-bound-sa-token\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:58.781611 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.781594 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc5dn\" (UniqueName: \"kubernetes.io/projected/13fd2c81-54ab-4afa-9afe-68094563bf08-kube-api-access-vc5dn\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.781663 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.781610 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcm8z\" (UniqueName: \"kubernetes.io/projected/8b60900b-7a0e-4039-97af-c8157ae19ca0-kube-api-access-fcm8z\") pod \"periodic-gathering-jr4s9-d8s6t\" (UID: \"8b60900b-7a0e-4039-97af-c8157ae19ca0\") " pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" Oct 30 21:49:58.781784 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.781766 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j58ln\" (UniqueName: \"kubernetes.io/projected/95097323-8907-45bc-8967-0de718adefbd-kube-api-access-j58ln\") pod \"cluster-monitoring-operator-884576c57-hmttl\" (UID: \"95097323-8907-45bc-8967-0de718adefbd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hmttl" Oct 30 21:49:58.782682 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.782664 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thhdq\" (UniqueName: \"kubernetes.io/projected/9df43c88-a01f-4ec7-a385-d5afcf6f9e6c-kube-api-access-thhdq\") pod \"kas-connection-checker-685895c7c9-k6p2v\" (UID: \"9df43c88-a01f-4ec7-a385-d5afcf6f9e6c\") " pod="kube-system/kas-connection-checker-685895c7c9-k6p2v" Oct 30 21:49:58.783402 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.783344 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxnj9\" (UniqueName: \"kubernetes.io/projected/3458688b-8142-492c-ab8b-bffec754c8c8-kube-api-access-hxnj9\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:58.783898 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.783876 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8258\" (UniqueName: \"kubernetes.io/projected/35a08208-7245-4e1b-9c0b-9e825a808365-kube-api-access-h8258\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-55pjz\" (UID: \"35a08208-7245-4e1b-9c0b-9e825a808365\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-55pjz" Oct 30 21:49:58.784112 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.784093 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3458688b-8142-492c-ab8b-bffec754c8c8-trusted-ca\") pod \"image-registry-6c68f8f5c6-wt2pc\" (UID: \"3458688b-8142-492c-ab8b-bffec754c8c8\") " pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:58.784701 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.784682 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-826wq\" (UniqueName: \"kubernetes.io/projected/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-kube-api-access-826wq\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:58.784739 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.784712 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hnzg\" (UniqueName: \"kubernetes.io/projected/d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2-kube-api-access-9hnzg\") pod \"service-ca-5fffc97bdf-mzhxr\" (UID: \"d1d7000a-8ca2-4dd3-9b87-a35afbfd2ad2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mzhxr" Oct 30 21:49:58.785389 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.785347 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6k4l\" (UniqueName: \"kubernetes.io/projected/a32b60f2-3000-4fe8-a928-955de8c76c82-kube-api-access-r6k4l\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.786476 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.786459 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-bound-sa-token\") pod \"image-registry-55fdbc6f6f-ngftj\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:49:58.791716 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.791695 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q7c2\" (UniqueName: \"kubernetes.io/projected/1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4-kube-api-access-7q7c2\") pod \"node-resolver-74zqz\" (UID: \"1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4\") " pod="openshift-dns/node-resolver-74zqz" Oct 30 21:49:58.794079 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.794062 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j92gp\" (UniqueName: \"kubernetes.io/projected/c0cece9c-2008-49ce-87de-4804d4e7de08-kube-api-access-j92gp\") pod \"insights-operator-7c4986d449-jrhw4\" (UID: \"c0cece9c-2008-49ce-87de-4804d4e7de08\") " pod="openshift-insights/insights-operator-7c4986d449-jrhw4" Oct 30 21:49:58.795683 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.795661 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hz67\" (UniqueName: \"kubernetes.io/projected/9161cdbd-c44f-4a8c-b918-107fc7001f3a-kube-api-access-9hz67\") pod \"iptables-alerter-gft82\" (UID: \"9161cdbd-c44f-4a8c-b918-107fc7001f3a\") " pod="openshift-network-operator/iptables-alerter-gft82" Oct 30 21:49:58.795778 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.795751 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw4xv\" (UniqueName: \"kubernetes.io/projected/b2f2c486-d661-4d59-981c-0b95a1f66fb4-kube-api-access-sw4xv\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:49:58.795910 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.795896 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1d430f7e-00f1-4b9e-9427-3e9971cfff31-kubelet-config\") pod \"global-pull-secret-syncer-lfq7z\" (UID: \"1d430f7e-00f1-4b9e-9427-3e9971cfff31\") " pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:49:58.795999 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.795916 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ceedf44e-88c1-49cf-901e-75558703c361-host\") pod \"node-ca-x4655\" (UID: \"ceedf44e-88c1-49cf-901e-75558703c361\") " pod="openshift-image-registry/node-ca-x4655" Oct 30 21:49:58.795999 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.795932 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-run-netns\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.795999 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.795950 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-kubernetes\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.795999 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.795965 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-slash\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.796177 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796011 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-kubernetes\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.796177 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796020 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1d430f7e-00f1-4b9e-9427-3e9971cfff31-kubelet-config\") pod \"global-pull-secret-syncer-lfq7z\" (UID: \"1d430f7e-00f1-4b9e-9427-3e9971cfff31\") " pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:49:58.796177 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796032 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-multus-socket-dir-parent\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.796177 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796017 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ceedf44e-88c1-49cf-901e-75558703c361-host\") pod \"node-ca-x4655\" (UID: \"ceedf44e-88c1-49cf-901e-75558703c361\") " pod="openshift-image-registry/node-ca-x4655" Oct 30 21:49:58.796177 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796060 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-run-k8s-cni-cncf-io\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.796177 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796063 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-multus-socket-dir-parent\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.796177 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796090 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-run-k8s-cni-cncf-io\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.796177 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796088 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-slash\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.796177 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796104 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-run-ovn\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.796177 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796124 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-run-ovn\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.796177 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796087 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-run-netns\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.796177 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796132 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-node-log\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.796177 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796154 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-node-log\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.796177 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796160 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-device-dir\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:49:58.796177 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796178 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-sysctl-d\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796186 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-device-dir\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796193 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-hostroot\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796209 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-etc-selinux\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796227 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-var-lib-cni-bin\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796242 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-run\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796258 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-hostroot\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796276 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-lib-modules\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796278 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-sysctl-d\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796304 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-run-openvswitch\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796309 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-var-lib-cni-bin\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796326 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9161cdbd-c44f-4a8c-b918-107fc7001f3a-host-slash\") pod \"iptables-alerter-gft82\" (UID: \"9161cdbd-c44f-4a8c-b918-107fc7001f3a\") " pod="openshift-network-operator/iptables-alerter-gft82" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796328 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-run\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796344 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-run-systemd\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796328 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-etc-selinux\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796346 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-lib-modules\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796357 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-run-openvswitch\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796374 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-run-systemd\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796379 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1d430f7e-00f1-4b9e-9427-3e9971cfff31-dbus\") pod \"global-pull-secret-syncer-lfq7z\" (UID: \"1d430f7e-00f1-4b9e-9427-3e9971cfff31\") " pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796396 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9161cdbd-c44f-4a8c-b918-107fc7001f3a-host-slash\") pod \"iptables-alerter-gft82\" (UID: \"9161cdbd-c44f-4a8c-b918-107fc7001f3a\") " pod="openshift-network-operator/iptables-alerter-gft82" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796457 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1d430f7e-00f1-4b9e-9427-3e9971cfff31-dbus\") pod \"global-pull-secret-syncer-lfq7z\" (UID: \"1d430f7e-00f1-4b9e-9427-3e9971cfff31\") " pod="kube-system/global-pull-secret-syncer-lfq7z" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796461 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-sys\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796491 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-run-multus-certs\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796494 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-sys\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796530 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-kubelet\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796565 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-run-multus-certs\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796564 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796594 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-kubelet\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796597 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-var-lib-kubelet\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796627 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-sys-fs\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796596 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796656 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-systemd\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796675 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-sys-fs\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796649 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-var-lib-kubelet\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796683 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-sysctl-conf\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796700 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-systemd\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796711 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-socket-dir\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796737 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-run-ovn-kubernetes\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796779 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-socket-dir\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796794 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-sysctl-conf\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796804 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-run-ovn-kubernetes\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796794 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-sysconfig\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796830 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-var-lib-kubelet\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796868 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-sysconfig\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.796865 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796870 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-var-lib-kubelet\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.796967 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-multus-cni-dir\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797213 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-multus-cni-dir\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797234 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-log-socket\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797262 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdhdv\" (UniqueName: \"kubernetes.io/projected/f71e935d-f091-4f16-820b-8d7fc632f003-kube-api-access-vdhdv\") pod \"migrator-66df5d496b-f2dgh\" (UID: \"f71e935d-f091-4f16-820b-8d7fc632f003\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-f2dgh" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797279 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-log-socket\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797285 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-cni-netd\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797264 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-cni-netd\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797310 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-crio-socket\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797327 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffa64e38-3f1b-49af-913f-139addefa359-cnibin\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797384 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5b2dbc17-c725-4893-896e-4dd2ec2dbc13-crio-socket\") pod \"insights-runtime-extractor-hkqxb\" (UID: \"5b2dbc17-c725-4893-896e-4dd2ec2dbc13\") " pod="openshift-insights/insights-runtime-extractor-hkqxb" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797428 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffa64e38-3f1b-49af-913f-139addefa359-cnibin\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797456 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-var-lib-cni-multus\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797484 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-modprobe-d\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797507 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-system-cni-dir\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797542 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-host-var-lib-cni-multus\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797561 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-registration-dir\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797578 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-etc-kubernetes\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797585 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-etc-modprobe-d\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797595 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-cni-bin\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797619 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-registration-dir\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797621 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797647 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffa64e38-3f1b-49af-913f-139addefa359-system-cni-dir\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797658 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-cni-bin\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797667 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b2f2c486-d661-4d59-981c-0b95a1f66fb4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cqwn8\" (UID: \"b2f2c486-d661-4d59-981c-0b95a1f66fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqwn8" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797622 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-etc-kubernetes\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797595 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-system-cni-dir\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797673 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffa64e38-3f1b-49af-913f-139addefa359-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797711 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffa64e38-3f1b-49af-913f-139addefa359-system-cni-dir\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797726 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffa64e38-3f1b-49af-913f-139addefa359-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2t6m\" (UID: \"ffa64e38-3f1b-49af-913f-139addefa359\") " pod="openshift-multus/multus-additional-cni-plugins-w2t6m" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797733 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-host\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797748 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-cnibin\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797766 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-multus-conf-dir\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797781 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a32b60f2-3000-4fe8-a928-955de8c76c82-host\") pod \"tuned-ld4hs\" (UID: \"a32b60f2-3000-4fe8-a928-955de8c76c82\") " pod="openshift-cluster-node-tuning-operator/tuned-ld4hs" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797785 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-systemd-units\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797810 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-systemd-units\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797816 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-run-netns\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797824 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-cnibin\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797840 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-host-run-netns\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797843 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-var-lib-openvswitch\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797863 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-etc-openvswitch\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797826 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13fd2c81-54ab-4afa-9afe-68094563bf08-multus-conf-dir\") pod \"multus-mmdsp\" (UID: \"13fd2c81-54ab-4afa-9afe-68094563bf08\") " pod="openshift-multus/multus-mmdsp" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797880 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4-hosts-file\") pod \"node-resolver-74zqz\" (UID: \"1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4\") " pod="openshift-dns/node-resolver-74zqz" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797887 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-etc-openvswitch\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797865 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7aa1b80f-115e-4708-bc98-88d13ad66860-var-lib-openvswitch\") pod \"ovnkube-node-vtgcr\" (UID: \"7aa1b80f-115e-4708-bc98-88d13ad66860\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.797952 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4-hosts-file\") pod \"node-resolver-74zqz\" (UID: \"1a96e3f0-b669-4a6b-a7fe-a9accbfc73b4\") " pod="openshift-dns/node-resolver-74zqz" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.798155 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2qx6\" (UniqueName: \"kubernetes.io/projected/bd3b421a-9313-4f15-b81f-900717685fd6-kube-api-access-j2qx6\") pod \"network-metrics-daemon-69j9v\" (UID: \"bd3b421a-9313-4f15-b81f-900717685fd6\") " pod="openshift-multus/network-metrics-daemon-69j9v" Oct 30 21:49:58.798415 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.798222 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f54bb\" (UniqueName: \"kubernetes.io/projected/6e73e56e-9619-4b97-bf20-a6291acf6355-kube-api-access-f54bb\") pod \"migrator-66df5d496b-27sld\" (UID: \"6e73e56e-9619-4b97-bf20-a6291acf6355\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-27sld" Oct 30 21:49:58.799754 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.799040 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8pl2\" (UniqueName: \"kubernetes.io/projected/ceedf44e-88c1-49cf-901e-75558703c361-kube-api-access-s8pl2\") pod \"node-ca-x4655\" (UID: \"ceedf44e-88c1-49cf-901e-75558703c361\") " pod="openshift-image-registry/node-ca-x4655" Oct 30 21:49:58.799754 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.799626 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67x9l\" (UniqueName: \"kubernetes.io/projected/226c41d5-de84-4950-978f-a1f0b4cbc319-kube-api-access-67x9l\") pod \"network-check-source-6f78d59f99-989z4\" (UID: \"226c41d5-de84-4950-978f-a1f0b4cbc319\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-989z4" Oct 30 21:49:58.939431 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.939394 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:49:58.957101 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:58.957080 6006 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:49:59.680568 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:59.680533 6006 generic.go:356] "Generic (PLEG): container finished" podID="11109f61-f4f0-4253-aacb-886c353a5772" containerID="f1a4144efc243977f35c360dbc48feae88935bc5e232cf4b5fc9e02cffdb1059" exitCode=0 Oct 30 21:49:59.680980 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:59.680605 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88kh6" event={"ID":"11109f61-f4f0-4253-aacb-886c353a5772","Type":"ContainerDied","Data":"f1a4144efc243977f35c360dbc48feae88935bc5e232cf4b5fc9e02cffdb1059"} Oct 30 21:49:59.680980 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:59.680957 6006 scope.go:122] "RemoveContainer" containerID="f1a4144efc243977f35c360dbc48feae88935bc5e232cf4b5fc9e02cffdb1059" Oct 30 21:49:59.682298 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:59.682272 6006 generic.go:356] "Generic (PLEG): container finished" podID="c0cece9c-2008-49ce-87de-4804d4e7de08" containerID="56c0ca7e9b2bbdaa9faf5371a0665bd874ecbb21d852d6bf8f04f7ad410787ea" exitCode=0 Oct 30 21:49:59.682396 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:59.682369 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-jrhw4" event={"ID":"c0cece9c-2008-49ce-87de-4804d4e7de08","Type":"ContainerDied","Data":"56c0ca7e9b2bbdaa9faf5371a0665bd874ecbb21d852d6bf8f04f7ad410787ea"} Oct 30 21:49:59.682988 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:59.682971 6006 scope.go:122] "RemoveContainer" containerID="56c0ca7e9b2bbdaa9faf5371a0665bd874ecbb21d852d6bf8f04f7ad410787ea" Oct 30 21:49:59.816650 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:59.816140 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:49:59.941414 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:49:59.941337 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:50:00.462039 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:00.462002 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:50:00.527946 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:00.527911 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55fdbc6f6f-ngftj"] Oct 30 21:50:00.690609 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:00.690569 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-jrhw4" event={"ID":"c0cece9c-2008-49ce-87de-4804d4e7de08","Type":"ContainerStarted","Data":"4451f2f547345b353482680b9cc196cdb2ad793f5c58371d9a22343c92402c90"} Oct 30 21:50:00.692816 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:00.692785 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88kh6" event={"ID":"11109f61-f4f0-4253-aacb-886c353a5772","Type":"ContainerStarted","Data":"bbc948e97142d53b34d4010debc4871b4ffa31ebee20f8c7eca73b3e47ff0800"} Oct 30 21:50:00.703924 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:00.703899 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-p8hw6" Oct 30 21:50:01.346989 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:01.346951 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-p4dp7-9bhtw"] Oct 30 21:50:01.368600 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:01.368209 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-p4dp7-9bhtw" Oct 30 21:50:01.405645 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:01.405560 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-p4dp7-9bhtw"] Oct 30 21:50:01.423658 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:01.423610 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa51475c-cd5b-4d46-891b-6d222a55a046-service-ca-bundle\") pod \"periodic-gathering-p4dp7-9bhtw\" (UID: \"aa51475c-cd5b-4d46-891b-6d222a55a046\") " pod="openshift-insights/periodic-gathering-p4dp7-9bhtw" Oct 30 21:50:01.423839 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:01.423676 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/aa51475c-cd5b-4d46-891b-6d222a55a046-archives-path\") pod \"periodic-gathering-p4dp7-9bhtw\" (UID: \"aa51475c-cd5b-4d46-891b-6d222a55a046\") " pod="openshift-insights/periodic-gathering-p4dp7-9bhtw" Oct 30 21:50:01.426987 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:01.426944 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa51475c-cd5b-4d46-891b-6d222a55a046-trusted-ca-bundle\") pod \"periodic-gathering-p4dp7-9bhtw\" (UID: \"aa51475c-cd5b-4d46-891b-6d222a55a046\") " pod="openshift-insights/periodic-gathering-p4dp7-9bhtw" Oct 30 21:50:01.427118 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:01.427028 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zzkq\" (UniqueName: \"kubernetes.io/projected/aa51475c-cd5b-4d46-891b-6d222a55a046-kube-api-access-7zzkq\") pod \"periodic-gathering-p4dp7-9bhtw\" (UID: \"aa51475c-cd5b-4d46-891b-6d222a55a046\") " pod="openshift-insights/periodic-gathering-p4dp7-9bhtw" Oct 30 21:50:01.528390 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:01.528342 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa51475c-cd5b-4d46-891b-6d222a55a046-trusted-ca-bundle\") pod \"periodic-gathering-p4dp7-9bhtw\" (UID: \"aa51475c-cd5b-4d46-891b-6d222a55a046\") " pod="openshift-insights/periodic-gathering-p4dp7-9bhtw" Oct 30 21:50:01.528572 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:01.528414 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zzkq\" (UniqueName: \"kubernetes.io/projected/aa51475c-cd5b-4d46-891b-6d222a55a046-kube-api-access-7zzkq\") pod \"periodic-gathering-p4dp7-9bhtw\" (UID: \"aa51475c-cd5b-4d46-891b-6d222a55a046\") " pod="openshift-insights/periodic-gathering-p4dp7-9bhtw" Oct 30 21:50:01.528572 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:01.528479 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa51475c-cd5b-4d46-891b-6d222a55a046-service-ca-bundle\") pod \"periodic-gathering-p4dp7-9bhtw\" (UID: \"aa51475c-cd5b-4d46-891b-6d222a55a046\") " pod="openshift-insights/periodic-gathering-p4dp7-9bhtw" Oct 30 21:50:01.528572 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:01.528511 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/aa51475c-cd5b-4d46-891b-6d222a55a046-archives-path\") pod \"periodic-gathering-p4dp7-9bhtw\" (UID: \"aa51475c-cd5b-4d46-891b-6d222a55a046\") " pod="openshift-insights/periodic-gathering-p4dp7-9bhtw" Oct 30 21:50:01.528983 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:01.528953 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/aa51475c-cd5b-4d46-891b-6d222a55a046-archives-path\") pod \"periodic-gathering-p4dp7-9bhtw\" (UID: \"aa51475c-cd5b-4d46-891b-6d222a55a046\") " pod="openshift-insights/periodic-gathering-p4dp7-9bhtw" Oct 30 21:50:01.529487 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:01.529462 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa51475c-cd5b-4d46-891b-6d222a55a046-service-ca-bundle\") pod \"periodic-gathering-p4dp7-9bhtw\" (UID: \"aa51475c-cd5b-4d46-891b-6d222a55a046\") " pod="openshift-insights/periodic-gathering-p4dp7-9bhtw" Oct 30 21:50:01.529614 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:01.529589 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa51475c-cd5b-4d46-891b-6d222a55a046-trusted-ca-bundle\") pod \"periodic-gathering-p4dp7-9bhtw\" (UID: \"aa51475c-cd5b-4d46-891b-6d222a55a046\") " pod="openshift-insights/periodic-gathering-p4dp7-9bhtw" Oct 30 21:50:01.584850 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:01.584815 6006 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:50:01.587684 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:01.587651 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zzkq\" (UniqueName: \"kubernetes.io/projected/aa51475c-cd5b-4d46-891b-6d222a55a046-kube-api-access-7zzkq\") pod \"periodic-gathering-p4dp7-9bhtw\" (UID: \"aa51475c-cd5b-4d46-891b-6d222a55a046\") " pod="openshift-insights/periodic-gathering-p4dp7-9bhtw" Oct 30 21:50:01.697485 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:01.697407 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-p4dp7-9bhtw" Oct 30 21:50:01.954649 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:01.954576 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:50:02.097136 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:02.097109 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-p4dp7-9bhtw"] Oct 30 21:50:02.100171 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:50:02.100135 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa51475c_cd5b_4d46_891b_6d222a55a046.slice/crio-a415b07497b6ecbbeb05f2df7a7c0b559bdb9ea4dcd3f13e00e1bd7ec6c4f30b WatchSource:0}: Error finding container a415b07497b6ecbbeb05f2df7a7c0b559bdb9ea4dcd3f13e00e1bd7ec6c4f30b: Status 404 returned error can't find the container with id a415b07497b6ecbbeb05f2df7a7c0b559bdb9ea4dcd3f13e00e1bd7ec6c4f30b Oct 30 21:50:02.705895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:02.705797 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-p4dp7-9bhtw" event={"ID":"aa51475c-cd5b-4d46-891b-6d222a55a046","Type":"ContainerStarted","Data":"f2b84575c3dad4bcf3d489862f792b865bbafc38e4c67a4aa8c2510ee2e09421"} Oct 30 21:50:02.705895 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:02.705860 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-p4dp7-9bhtw" event={"ID":"aa51475c-cd5b-4d46-891b-6d222a55a046","Type":"ContainerStarted","Data":"a415b07497b6ecbbeb05f2df7a7c0b559bdb9ea4dcd3f13e00e1bd7ec6c4f30b"} Oct 30 21:50:02.816657 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:02.816588 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-p4dp7-9bhtw" podStartSLOduration=1.816570266 podStartE2EDuration="1.816570266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:02.815705655 +0000 UTC m=+5.367881954" watchObservedRunningTime="2026-10-30 21:50:02.816570266 +0000 UTC m=+5.368746568" Oct 30 21:50:03.655564 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:03.655480 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:50:03.655749 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:03.655645 6006 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:50:03.657004 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:03.656983 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-76684dbbf-8mwh7" Oct 30 21:50:03.712941 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:03.712907 6006 generic.go:356] "Generic (PLEG): container finished" podID="35a08208-7245-4e1b-9c0b-9e825a808365" containerID="eb52ed220b3b8f921078ea044a235c08b4a3816a746a5ef554b68224eb7ad308" exitCode=0 Oct 30 21:50:03.713479 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:03.712999 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-55pjz" event={"ID":"35a08208-7245-4e1b-9c0b-9e825a808365","Type":"ContainerDied","Data":"eb52ed220b3b8f921078ea044a235c08b4a3816a746a5ef554b68224eb7ad308"} Oct 30 21:50:03.713874 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:03.713856 6006 scope.go:122] "RemoveContainer" containerID="eb52ed220b3b8f921078ea044a235c08b4a3816a746a5ef554b68224eb7ad308" Oct 30 21:50:04.392094 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:04.392055 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" Oct 30 21:50:04.414903 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:04.414869 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:50:04.415141 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:04.415123 6006 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:50:04.435988 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:04.435958 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6gzkm" Oct 30 21:50:04.724980 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:04.724893 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-55pjz" event={"ID":"35a08208-7245-4e1b-9c0b-9e825a808365","Type":"ContainerStarted","Data":"fcda829518d3b5f64c1b0ca6adcd704cdd956fd7096238a5546bb1865815f498"} Oct 30 21:50:05.142941 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.142898 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wprgj"] Oct 30 21:50:05.146953 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.146914 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.243225 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.243190 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:50:05.243435 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.243327 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:50:05.243435 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.243421 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dhl5t\"" Oct 30 21:50:05.243563 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.243510 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:50:05.243617 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.243582 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:50:05.264803 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.264749 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsn6j\" (UniqueName: \"kubernetes.io/projected/94550423-24fa-4ec9-b44d-9cd54b08f75d-kube-api-access-dsn6j\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.264984 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.264828 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94550423-24fa-4ec9-b44d-9cd54b08f75d-sys\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.264984 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.264859 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/94550423-24fa-4ec9-b44d-9cd54b08f75d-root\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.264984 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.264891 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/94550423-24fa-4ec9-b44d-9cd54b08f75d-node-exporter-wtmp\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.264984 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.264945 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/94550423-24fa-4ec9-b44d-9cd54b08f75d-node-exporter-textfile\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.265204 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.264988 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/94550423-24fa-4ec9-b44d-9cd54b08f75d-node-exporter-tls\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.265204 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.265033 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/94550423-24fa-4ec9-b44d-9cd54b08f75d-node-exporter-accelerators-collector-config\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.265204 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.265083 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94550423-24fa-4ec9-b44d-9cd54b08f75d-metrics-client-ca\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.265204 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.265109 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94550423-24fa-4ec9-b44d-9cd54b08f75d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.366582 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.366540 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/94550423-24fa-4ec9-b44d-9cd54b08f75d-node-exporter-textfile\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.366773 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.366606 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/94550423-24fa-4ec9-b44d-9cd54b08f75d-node-exporter-tls\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.366773 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.366648 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/94550423-24fa-4ec9-b44d-9cd54b08f75d-node-exporter-accelerators-collector-config\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.366773 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.366691 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94550423-24fa-4ec9-b44d-9cd54b08f75d-metrics-client-ca\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.366773 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.366719 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94550423-24fa-4ec9-b44d-9cd54b08f75d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.366773 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.366762 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsn6j\" (UniqueName: \"kubernetes.io/projected/94550423-24fa-4ec9-b44d-9cd54b08f75d-kube-api-access-dsn6j\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.367056 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.366787 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94550423-24fa-4ec9-b44d-9cd54b08f75d-sys\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.367056 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.366815 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/94550423-24fa-4ec9-b44d-9cd54b08f75d-root\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.367056 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.366842 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/94550423-24fa-4ec9-b44d-9cd54b08f75d-node-exporter-wtmp\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.367056 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.366998 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/94550423-24fa-4ec9-b44d-9cd54b08f75d-node-exporter-wtmp\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.367056 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.367051 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94550423-24fa-4ec9-b44d-9cd54b08f75d-sys\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.367329 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.367088 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/94550423-24fa-4ec9-b44d-9cd54b08f75d-root\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.367421 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.367390 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/94550423-24fa-4ec9-b44d-9cd54b08f75d-node-exporter-accelerators-collector-config\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.367510 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.367104 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/94550423-24fa-4ec9-b44d-9cd54b08f75d-node-exporter-textfile\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.367685 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.367519 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94550423-24fa-4ec9-b44d-9cd54b08f75d-metrics-client-ca\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.369861 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.369835 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/94550423-24fa-4ec9-b44d-9cd54b08f75d-node-exporter-tls\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.370147 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.370119 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94550423-24fa-4ec9-b44d-9cd54b08f75d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.438385 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.430346 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsn6j\" (UniqueName: \"kubernetes.io/projected/94550423-24fa-4ec9-b44d-9cd54b08f75d-kube-api-access-dsn6j\") pod \"node-exporter-wprgj\" (UID: \"94550423-24fa-4ec9-b44d-9cd54b08f75d\") " pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.466095 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.466061 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wprgj" Oct 30 21:50:05.479759 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:50:05.479724 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94550423_24fa_4ec9_b44d_9cd54b08f75d.slice/crio-48a20f44e34bb621f82e962db823e770897efe53eb4773fdde299d46b19a9912 WatchSource:0}: Error finding container 48a20f44e34bb621f82e962db823e770897efe53eb4773fdde299d46b19a9912: Status 404 returned error can't find the container with id 48a20f44e34bb621f82e962db823e770897efe53eb4773fdde299d46b19a9912 Oct 30 21:50:05.731037 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:05.730949 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wprgj" event={"ID":"94550423-24fa-4ec9-b44d-9cd54b08f75d","Type":"ContainerStarted","Data":"48a20f44e34bb621f82e962db823e770897efe53eb4773fdde299d46b19a9912"} Oct 30 21:50:06.472535 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:06.472494 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-55fdbc6f6f-ngftj_84bfdb37-6346-4384-8d97-d43aba6f03fe/registry/0.log" Oct 30 21:50:06.584180 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:06.584149 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6c68f8f5c6-wt2pc_3458688b-8142-492c-ab8b-bffec754c8c8/registry/0.log" Oct 30 21:50:06.793268 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:06.793155 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x4655_ceedf44e-88c1-49cf-901e-75558703c361/node-ca/0.log" Oct 30 21:50:07.010631 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:07.010540 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76684dbbf-8mwh7_11ba3d2b-2d12-458c-9984-1dd82c275e10/router/0.log" Oct 30 21:50:07.152024 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:07.151997 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b6fbt_c7fe5efb-7029-41b8-8bbe-f37f49410f27/serve-healthcheck-canary/0.log" Oct 30 21:50:07.674825 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:07.674794 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-w7pwq_fc3ff956-6466-46e3-b734-13a3ac148d36/cluster-samples-operator/0.log" Oct 30 21:50:07.743775 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:07.743722 6006 generic.go:356] "Generic (PLEG): container finished" podID="94550423-24fa-4ec9-b44d-9cd54b08f75d" containerID="b6a4c2d7012340582ade7e41153633bc198da10166c14f1f4255490394c7459b" exitCode=0 Oct 30 21:50:07.743946 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:07.743849 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wprgj" event={"ID":"94550423-24fa-4ec9-b44d-9cd54b08f75d","Type":"ContainerDied","Data":"b6a4c2d7012340582ade7e41153633bc198da10166c14f1f4255490394c7459b"} Oct 30 21:50:07.874100 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:07.874000 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-w7pwq_fc3ff956-6466-46e3-b734-13a3ac148d36/cluster-samples-operator-watch/0.log" Oct 30 21:50:08.461107 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:08.461077 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bc452" Oct 30 21:50:08.465340 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:08.465311 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bc452" Oct 30 21:50:08.752900 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:08.752806 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wprgj" event={"ID":"94550423-24fa-4ec9-b44d-9cd54b08f75d","Type":"ContainerStarted","Data":"56d5f80040e63fdb63f0dda03f026def7d2cc5f47bd79c5142e2a97daf05cff8"} Oct 30 21:50:08.752900 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:08.752873 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wprgj" event={"ID":"94550423-24fa-4ec9-b44d-9cd54b08f75d","Type":"ContainerStarted","Data":"636a484422e7955abb577378cfd4dc25754c0764a8c0160fd34a684c968a0b8f"} Oct 30 21:50:08.868547 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:08.868492 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wprgj" podStartSLOduration=2.124128881 podStartE2EDuration="3.868476974s" totalImagesPullingTime="1.662103045s" totalInitContainerRuntime="82.245048ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:50:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:08.867746098 +0000 UTC m=+11.419922398" watchObservedRunningTime="2026-10-30 21:50:08.868476974 +0000 UTC m=+11.420653264" Oct 30 21:50:09.823260 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:09.821942 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c68f8f5c6-wt2pc" Oct 30 21:50:10.697976 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:10.697944 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:50:16.794531 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:16.794494 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-tqt4b" event={"ID":"ce930258-60d9-4988-913c-2f3cb78eac8d","Type":"ContainerStarted","Data":"b05e0b72824afe2382e53d58fe5b08173b69c7af8e885b6da0bb0576abf0dea0"} Oct 30 21:50:16.795092 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:16.794820 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-tqt4b" Oct 30 21:50:16.812653 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:16.812528 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-tqt4b" Oct 30 21:50:19.453096 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:19.453055 6006 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-929rz_64086350-55f6-4da5-a871-3f23c1755371/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:21.058705 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:21.058663 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-jr4s9-d8s6t"] Oct 30 21:50:21.059156 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:21.058888 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" podUID="8b60900b-7a0e-4039-97af-c8157ae19ca0" containerName="insights-gathering" containerID="cri-o://c1443ed6373d5d19747e0f15060e8f266bfc59abf7fb41a8cc83e4c58ed79c16" gracePeriod=30 Oct 30 21:50:21.387838 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:21.387801 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" Oct 30 21:50:21.418920 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:21.418881 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b60900b-7a0e-4039-97af-c8157ae19ca0-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b60900b-7a0e-4039-97af-c8157ae19ca0-service-ca-bundle\") pod \"8b60900b-7a0e-4039-97af-c8157ae19ca0\" (UID: \"8b60900b-7a0e-4039-97af-c8157ae19ca0\") " Oct 30 21:50:21.419116 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:21.418967 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b60900b-7a0e-4039-97af-c8157ae19ca0-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b60900b-7a0e-4039-97af-c8157ae19ca0-trusted-ca-bundle\") pod \"8b60900b-7a0e-4039-97af-c8157ae19ca0\" (UID: \"8b60900b-7a0e-4039-97af-c8157ae19ca0\") " Oct 30 21:50:21.419116 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:21.419005 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b60900b-7a0e-4039-97af-c8157ae19ca0-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8b60900b-7a0e-4039-97af-c8157ae19ca0-archives-path\") pod \"8b60900b-7a0e-4039-97af-c8157ae19ca0\" (UID: \"8b60900b-7a0e-4039-97af-c8157ae19ca0\") " Oct 30 21:50:21.419116 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:21.419082 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b60900b-7a0e-4039-97af-c8157ae19ca0-kube-api-access-fcm8z\" (UniqueName: \"kubernetes.io/projected/8b60900b-7a0e-4039-97af-c8157ae19ca0-kube-api-access-fcm8z\") pod \"8b60900b-7a0e-4039-97af-c8157ae19ca0\" (UID: \"8b60900b-7a0e-4039-97af-c8157ae19ca0\") " Oct 30 21:50:21.419518 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:21.419450 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b60900b-7a0e-4039-97af-c8157ae19ca0-trusted-ca-bundle" pod "8b60900b-7a0e-4039-97af-c8157ae19ca0" (UID: "8b60900b-7a0e-4039-97af-c8157ae19ca0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:21.419518 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:21.419464 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b60900b-7a0e-4039-97af-c8157ae19ca0-service-ca-bundle" pod "8b60900b-7a0e-4039-97af-c8157ae19ca0" (UID: "8b60900b-7a0e-4039-97af-c8157ae19ca0"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:21.419518 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:21.419507 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b60900b-7a0e-4039-97af-c8157ae19ca0-archives-path" pod "8b60900b-7a0e-4039-97af-c8157ae19ca0" (UID: "8b60900b-7a0e-4039-97af-c8157ae19ca0"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:21.421863 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:21.421835 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b60900b-7a0e-4039-97af-c8157ae19ca0-kube-api-access-fcm8z" pod "8b60900b-7a0e-4039-97af-c8157ae19ca0" (UID: "8b60900b-7a0e-4039-97af-c8157ae19ca0"). InnerVolumeSpecName "kube-api-access-fcm8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:21.519926 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:21.519886 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcm8z\" (UniqueName: \"kubernetes.io/projected/8b60900b-7a0e-4039-97af-c8157ae19ca0-kube-api-access-fcm8z\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:21.519926 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:21.519921 6006 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b60900b-7a0e-4039-97af-c8157ae19ca0-service-ca-bundle\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:21.520185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:21.519938 6006 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b60900b-7a0e-4039-97af-c8157ae19ca0-trusted-ca-bundle\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:21.520185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:21.519954 6006 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8b60900b-7a0e-4039-97af-c8157ae19ca0-archives-path\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:21.829935 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:21.829893 6006 generic.go:356] "Generic (PLEG): container finished" podID="8b60900b-7a0e-4039-97af-c8157ae19ca0" containerID="c1443ed6373d5d19747e0f15060e8f266bfc59abf7fb41a8cc83e4c58ed79c16" exitCode=2 Oct 30 21:50:21.830102 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:21.829947 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" Oct 30 21:50:21.830102 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:21.829989 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" event={"ID":"8b60900b-7a0e-4039-97af-c8157ae19ca0","Type":"ContainerDied","Data":"c1443ed6373d5d19747e0f15060e8f266bfc59abf7fb41a8cc83e4c58ed79c16"} Oct 30 21:50:21.830102 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:21.830065 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jr4s9-d8s6t" event={"ID":"8b60900b-7a0e-4039-97af-c8157ae19ca0","Type":"ContainerDied","Data":"9ea7a7f39436bd95eb87aaa921bf467e4ccb983ecc7b9cb726badd8d5ea2b22e"} Oct 30 21:50:21.830102 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:21.830093 6006 scope.go:122] "RemoveContainer" containerID="c1443ed6373d5d19747e0f15060e8f266bfc59abf7fb41a8cc83e4c58ed79c16" Oct 30 21:50:21.841061 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:21.841033 6006 scope.go:122] "RemoveContainer" containerID="c1443ed6373d5d19747e0f15060e8f266bfc59abf7fb41a8cc83e4c58ed79c16" Oct 30 21:50:21.841630 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:50:21.841594 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1443ed6373d5d19747e0f15060e8f266bfc59abf7fb41a8cc83e4c58ed79c16\": container with ID starting with c1443ed6373d5d19747e0f15060e8f266bfc59abf7fb41a8cc83e4c58ed79c16 not found: ID does not exist" containerID="c1443ed6373d5d19747e0f15060e8f266bfc59abf7fb41a8cc83e4c58ed79c16" Oct 30 21:50:21.841759 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:21.841642 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1443ed6373d5d19747e0f15060e8f266bfc59abf7fb41a8cc83e4c58ed79c16"} err="failed to get container status \"c1443ed6373d5d19747e0f15060e8f266bfc59abf7fb41a8cc83e4c58ed79c16\": rpc error: code = NotFound desc = could not find container \"c1443ed6373d5d19747e0f15060e8f266bfc59abf7fb41a8cc83e4c58ed79c16\": container with ID starting with c1443ed6373d5d19747e0f15060e8f266bfc59abf7fb41a8cc83e4c58ed79c16 not found: ID does not exist" Oct 30 21:50:21.985388 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:21.985306 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-jr4s9-d8s6t"] Oct 30 21:50:22.032468 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:22.032421 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-jr4s9-d8s6t"] Oct 30 21:50:22.597747 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:22.597715 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vtgcr" Oct 30 21:50:23.603290 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:23.603236 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b60900b-7a0e-4039-97af-c8157ae19ca0" path="/var/lib/kubelet/pods/8b60900b-7a0e-4039-97af-c8157ae19ca0/volumes" Oct 30 21:50:25.715316 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:25.715237 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" podUID="84bfdb37-6346-4384-8d97-d43aba6f03fe" containerName="registry" containerID="cri-o://0fe6220e95f2245e721938f428ebed95acae6b241b6f700a621c9c69e676f8be" gracePeriod=30 Oct 30 21:50:25.859634 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:25.859593 6006 generic.go:356] "Generic (PLEG): container finished" podID="84bfdb37-6346-4384-8d97-d43aba6f03fe" containerID="0fe6220e95f2245e721938f428ebed95acae6b241b6f700a621c9c69e676f8be" exitCode=0 Oct 30 21:50:25.859808 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:25.859672 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" event={"ID":"84bfdb37-6346-4384-8d97-d43aba6f03fe","Type":"ContainerDied","Data":"0fe6220e95f2245e721938f428ebed95acae6b241b6f700a621c9c69e676f8be"} Oct 30 21:50:26.008873 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.008836 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:50:26.020546 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.020519 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-929rz" Oct 30 21:50:26.176834 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.176800 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-certificates\") pod \"84bfdb37-6346-4384-8d97-d43aba6f03fe\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " Oct 30 21:50:26.177017 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.176851 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84bfdb37-6346-4384-8d97-d43aba6f03fe-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/84bfdb37-6346-4384-8d97-d43aba6f03fe-ca-trust-extracted\") pod \"84bfdb37-6346-4384-8d97-d43aba6f03fe\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " Oct 30 21:50:26.177017 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.176877 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84bfdb37-6346-4384-8d97-d43aba6f03fe-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/84bfdb37-6346-4384-8d97-d43aba6f03fe-image-registry-private-configuration\") pod \"84bfdb37-6346-4384-8d97-d43aba6f03fe\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " Oct 30 21:50:26.177017 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.176982 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84bfdb37-6346-4384-8d97-d43aba6f03fe-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/84bfdb37-6346-4384-8d97-d43aba6f03fe-installation-pull-secrets\") pod \"84bfdb37-6346-4384-8d97-d43aba6f03fe\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " Oct 30 21:50:26.177187 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.177038 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-bound-sa-token\") pod \"84bfdb37-6346-4384-8d97-d43aba6f03fe\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " Oct 30 21:50:26.177187 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.177071 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84bfdb37-6346-4384-8d97-d43aba6f03fe-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/84bfdb37-6346-4384-8d97-d43aba6f03fe-trusted-ca\") pod \"84bfdb37-6346-4384-8d97-d43aba6f03fe\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " Oct 30 21:50:26.177187 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.177104 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-kube-api-access-vss2v\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-kube-api-access-vss2v\") pod \"84bfdb37-6346-4384-8d97-d43aba6f03fe\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " Oct 30 21:50:26.177386 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.177222 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls\") pod \"84bfdb37-6346-4384-8d97-d43aba6f03fe\" (UID: \"84bfdb37-6346-4384-8d97-d43aba6f03fe\") " Oct 30 21:50:26.177457 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.177380 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-certificates" pod "84bfdb37-6346-4384-8d97-d43aba6f03fe" (UID: "84bfdb37-6346-4384-8d97-d43aba6f03fe"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:26.177659 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.177635 6006 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-certificates\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:26.177766 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.177666 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84bfdb37-6346-4384-8d97-d43aba6f03fe-trusted-ca" pod "84bfdb37-6346-4384-8d97-d43aba6f03fe" (UID: "84bfdb37-6346-4384-8d97-d43aba6f03fe"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:26.180002 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.179966 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84bfdb37-6346-4384-8d97-d43aba6f03fe-image-registry-private-configuration" pod "84bfdb37-6346-4384-8d97-d43aba6f03fe" (UID: "84bfdb37-6346-4384-8d97-d43aba6f03fe"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:26.180114 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.180007 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls" pod "84bfdb37-6346-4384-8d97-d43aba6f03fe" (UID: "84bfdb37-6346-4384-8d97-d43aba6f03fe"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:26.180114 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.180060 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84bfdb37-6346-4384-8d97-d43aba6f03fe-installation-pull-secrets" pod "84bfdb37-6346-4384-8d97-d43aba6f03fe" (UID: "84bfdb37-6346-4384-8d97-d43aba6f03fe"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:26.180243 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.180205 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-bound-sa-token" pod "84bfdb37-6346-4384-8d97-d43aba6f03fe" (UID: "84bfdb37-6346-4384-8d97-d43aba6f03fe"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:26.180335 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.180311 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-kube-api-access-vss2v" pod "84bfdb37-6346-4384-8d97-d43aba6f03fe" (UID: "84bfdb37-6346-4384-8d97-d43aba6f03fe"). InnerVolumeSpecName "kube-api-access-vss2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:26.185311 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.185241 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84bfdb37-6346-4384-8d97-d43aba6f03fe-ca-trust-extracted" pod "84bfdb37-6346-4384-8d97-d43aba6f03fe" (UID: "84bfdb37-6346-4384-8d97-d43aba6f03fe"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:26.278880 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.278790 6006 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-registry-tls\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:26.278880 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.278826 6006 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/84bfdb37-6346-4384-8d97-d43aba6f03fe-ca-trust-extracted\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:26.278880 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.278837 6006 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/84bfdb37-6346-4384-8d97-d43aba6f03fe-image-registry-private-configuration\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:26.278880 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.278847 6006 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/84bfdb37-6346-4384-8d97-d43aba6f03fe-installation-pull-secrets\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:26.278880 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.278857 6006 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-bound-sa-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:26.278880 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.278866 6006 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/84bfdb37-6346-4384-8d97-d43aba6f03fe-trusted-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:26.278880 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.278876 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vss2v\" (UniqueName: \"kubernetes.io/projected/84bfdb37-6346-4384-8d97-d43aba6f03fe-kube-api-access-vss2v\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:26.865147 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.865105 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" event={"ID":"84bfdb37-6346-4384-8d97-d43aba6f03fe","Type":"ContainerDied","Data":"4d9c6673201363e110fb9941032ebf8c83fc19e1fe99fe2539d7b22f07db02c6"} Oct 30 21:50:26.865147 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.865144 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55fdbc6f6f-ngftj" Oct 30 21:50:26.865645 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:26.865182 6006 scope.go:122] "RemoveContainer" containerID="0fe6220e95f2245e721938f428ebed95acae6b241b6f700a621c9c69e676f8be" Oct 30 21:50:27.069001 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:27.068963 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55fdbc6f6f-ngftj"] Oct 30 21:50:27.116183 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:27.116085 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-55fdbc6f6f-ngftj"] Oct 30 21:50:27.600718 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:50:27.600679 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84bfdb37-6346-4384-8d97-d43aba6f03fe" path="/var/lib/kubelet/pods/84bfdb37-6346-4384-8d97-d43aba6f03fe/volumes" Oct 30 21:50:27.909473 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:50:27.909372 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 21:50:57.909537 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:50:57.909444 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 21:51:27.908513 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:51:27.908476 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 21:51:30.116600 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:51:30.116564 6006 generic.go:356] "Generic (PLEG): container finished" podID="aa51475c-cd5b-4d46-891b-6d222a55a046" containerID="f2b84575c3dad4bcf3d489862f792b865bbafc38e4c67a4aa8c2510ee2e09421" exitCode=0 Oct 30 21:51:30.116999 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:51:30.116643 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-p4dp7-9bhtw" event={"ID":"aa51475c-cd5b-4d46-891b-6d222a55a046","Type":"ContainerDied","Data":"f2b84575c3dad4bcf3d489862f792b865bbafc38e4c67a4aa8c2510ee2e09421"} Oct 30 21:51:31.273339 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:51:31.273313 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-p4dp7-9bhtw" Oct 30 21:51:31.328694 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:51:31.328660 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa51475c-cd5b-4d46-891b-6d222a55a046-kube-api-access-7zzkq\" (UniqueName: \"kubernetes.io/projected/aa51475c-cd5b-4d46-891b-6d222a55a046-kube-api-access-7zzkq\") pod \"aa51475c-cd5b-4d46-891b-6d222a55a046\" (UID: \"aa51475c-cd5b-4d46-891b-6d222a55a046\") " Oct 30 21:51:31.328841 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:51:31.328703 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa51475c-cd5b-4d46-891b-6d222a55a046-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa51475c-cd5b-4d46-891b-6d222a55a046-service-ca-bundle\") pod \"aa51475c-cd5b-4d46-891b-6d222a55a046\" (UID: \"aa51475c-cd5b-4d46-891b-6d222a55a046\") " Oct 30 21:51:31.328841 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:51:31.328771 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa51475c-cd5b-4d46-891b-6d222a55a046-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa51475c-cd5b-4d46-891b-6d222a55a046-trusted-ca-bundle\") pod \"aa51475c-cd5b-4d46-891b-6d222a55a046\" (UID: \"aa51475c-cd5b-4d46-891b-6d222a55a046\") " Oct 30 21:51:31.329179 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:51:31.329149 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa51475c-cd5b-4d46-891b-6d222a55a046-service-ca-bundle" pod "aa51475c-cd5b-4d46-891b-6d222a55a046" (UID: "aa51475c-cd5b-4d46-891b-6d222a55a046"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:31.329179 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:51:31.329169 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa51475c-cd5b-4d46-891b-6d222a55a046-trusted-ca-bundle" pod "aa51475c-cd5b-4d46-891b-6d222a55a046" (UID: "aa51475c-cd5b-4d46-891b-6d222a55a046"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:31.330897 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:51:31.330876 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa51475c-cd5b-4d46-891b-6d222a55a046-kube-api-access-7zzkq" pod "aa51475c-cd5b-4d46-891b-6d222a55a046" (UID: "aa51475c-cd5b-4d46-891b-6d222a55a046"). InnerVolumeSpecName "kube-api-access-7zzkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:31.429583 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:51:31.429494 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa51475c-cd5b-4d46-891b-6d222a55a046-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/aa51475c-cd5b-4d46-891b-6d222a55a046-archives-path\") pod \"aa51475c-cd5b-4d46-891b-6d222a55a046\" (UID: \"aa51475c-cd5b-4d46-891b-6d222a55a046\") " Oct 30 21:51:31.429732 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:51:31.429632 6006 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa51475c-cd5b-4d46-891b-6d222a55a046-trusted-ca-bundle\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:31.429732 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:51:31.429644 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zzkq\" (UniqueName: \"kubernetes.io/projected/aa51475c-cd5b-4d46-891b-6d222a55a046-kube-api-access-7zzkq\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:31.429732 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:51:31.429653 6006 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa51475c-cd5b-4d46-891b-6d222a55a046-service-ca-bundle\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:31.429827 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:51:31.429797 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa51475c-cd5b-4d46-891b-6d222a55a046-archives-path" pod "aa51475c-cd5b-4d46-891b-6d222a55a046" (UID: "aa51475c-cd5b-4d46-891b-6d222a55a046"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:31.530433 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:51:31.530387 6006 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/aa51475c-cd5b-4d46-891b-6d222a55a046-archives-path\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:51:32.126986 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:51:32.126955 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-p4dp7-9bhtw" Oct 30 21:51:32.127177 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:51:32.126960 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-p4dp7-9bhtw" event={"ID":"aa51475c-cd5b-4d46-891b-6d222a55a046","Type":"ContainerDied","Data":"a415b07497b6ecbbeb05f2df7a7c0b559bdb9ea4dcd3f13e00e1bd7ec6c4f30b"} Oct 30 21:51:32.127177 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:51:32.127084 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a415b07497b6ecbbeb05f2df7a7c0b559bdb9ea4dcd3f13e00e1bd7ec6c4f30b" Oct 30 21:51:57.909360 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:51:57.909278 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 21:52:27.909407 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:52:27.909366 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 21:52:57.908824 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:52:57.908785 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 21:53:27.908664 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:53:27.908619 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 21:53:57.908441 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:53:57.908403 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 21:54:27.908924 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:54:27.908888 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 21:54:57.620686 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:54:57.620601 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 21:54:57.623550 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:54:57.623525 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 21:54:57.909834 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:54:57.909740 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 21:55:27.909697 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:55:27.909652 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 21:55:57.909186 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:55:57.909141 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 21:56:27.909995 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:56:27.909896 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 21:56:33.299005 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.298957 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj"] Oct 30 21:56:33.299557 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.299459 6006 cpu_manager.go:414] "containerMap: removing container" podUID="8b60900b-7a0e-4039-97af-c8157ae19ca0" containerName="insights-gathering" Oct 30 21:56:33.299557 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.299476 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b60900b-7a0e-4039-97af-c8157ae19ca0" containerName="insights-gathering" Oct 30 21:56:33.299557 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.299498 6006 cpu_manager.go:414] "containerMap: removing container" podUID="84bfdb37-6346-4384-8d97-d43aba6f03fe" containerName="registry" Oct 30 21:56:33.299557 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.299507 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84bfdb37-6346-4384-8d97-d43aba6f03fe" containerName="registry" Oct 30 21:56:33.299557 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.299520 6006 cpu_manager.go:414] "containerMap: removing container" podUID="aa51475c-cd5b-4d46-891b-6d222a55a046" containerName="insights-gathering" Oct 30 21:56:33.299557 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.299528 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa51475c-cd5b-4d46-891b-6d222a55a046" containerName="insights-gathering" Oct 30 21:56:33.299792 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.299610 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b60900b-7a0e-4039-97af-c8157ae19ca0" containerName="insights-gathering" Oct 30 21:56:33.299792 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.299625 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="84bfdb37-6346-4384-8d97-d43aba6f03fe" containerName="registry" Oct 30 21:56:33.299792 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.299633 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa51475c-cd5b-4d46-891b-6d222a55a046" containerName="insights-gathering" Oct 30 21:56:33.302840 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.302821 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.352535 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.352499 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 21:56:33.372492 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.372461 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v8cwr\"" Oct 30 21:56:33.400795 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.400760 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj"] Oct 30 21:56:33.417024 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.416982 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.417024 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.417028 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.417239 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.417048 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.417239 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.417175 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9jjq\" (UniqueName: \"kubernetes.io/projected/b77193b8-0b50-4d53-beb9-619be2ea30b5-kube-api-access-r9jjq\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.417239 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.417215 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.417359 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.417283 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.417359 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.417322 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.417359 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.417353 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.417532 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.417496 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b77193b8-0b50-4d53-beb9-619be2ea30b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.518950 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.518904 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b77193b8-0b50-4d53-beb9-619be2ea30b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.519158 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.518968 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.519158 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.518994 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.519158 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.519016 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.519158 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.519045 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9jjq\" (UniqueName: \"kubernetes.io/projected/b77193b8-0b50-4d53-beb9-619be2ea30b5-kube-api-access-r9jjq\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.519158 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.519083 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.519158 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.519109 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.519504 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.519407 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.519504 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.519467 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.519662 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.519499 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.519662 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.519646 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.519768 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.519720 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.519894 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.519871 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b77193b8-0b50-4d53-beb9-619be2ea30b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.520052 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.520026 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.521609 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.521586 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.521759 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.521740 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.578672 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.578592 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.578797 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.578707 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9jjq\" (UniqueName: \"kubernetes.io/projected/b77193b8-0b50-4d53-beb9-619be2ea30b5-kube-api-access-r9jjq\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gkxzj\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.602048 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.602016 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4"] Oct 30 21:56:33.606458 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.606438 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.615234 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.615206 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:33.658920 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.658883 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-sg9mj\"" Oct 30 21:56:33.665475 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.665382 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4"] Oct 30 21:56:33.721385 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.721345 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/989395fe-e528-4e51-96aa-e4b2a89e3b6f-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.721581 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.721399 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/989395fe-e528-4e51-96aa-e4b2a89e3b6f-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.721581 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.721499 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/989395fe-e528-4e51-96aa-e4b2a89e3b6f-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.721581 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.721534 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/989395fe-e528-4e51-96aa-e4b2a89e3b6f-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.721581 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.721560 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7br6d\" (UniqueName: \"kubernetes.io/projected/989395fe-e528-4e51-96aa-e4b2a89e3b6f-kube-api-access-7br6d\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.721748 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.721674 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/989395fe-e528-4e51-96aa-e4b2a89e3b6f-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.721748 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.721733 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/989395fe-e528-4e51-96aa-e4b2a89e3b6f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.721823 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.721794 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/989395fe-e528-4e51-96aa-e4b2a89e3b6f-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.721875 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.721859 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/989395fe-e528-4e51-96aa-e4b2a89e3b6f-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.823219 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.823176 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/989395fe-e528-4e51-96aa-e4b2a89e3b6f-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.823219 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.823222 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/989395fe-e528-4e51-96aa-e4b2a89e3b6f-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.823508 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.823242 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/989395fe-e528-4e51-96aa-e4b2a89e3b6f-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.823508 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.823297 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7br6d\" (UniqueName: \"kubernetes.io/projected/989395fe-e528-4e51-96aa-e4b2a89e3b6f-kube-api-access-7br6d\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.823508 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.823317 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/989395fe-e528-4e51-96aa-e4b2a89e3b6f-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.823508 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.823335 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/989395fe-e528-4e51-96aa-e4b2a89e3b6f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.823508 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.823353 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/989395fe-e528-4e51-96aa-e4b2a89e3b6f-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.823508 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.823402 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/989395fe-e528-4e51-96aa-e4b2a89e3b6f-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.823508 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.823449 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/989395fe-e528-4e51-96aa-e4b2a89e3b6f-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.823881 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.823659 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/989395fe-e528-4e51-96aa-e4b2a89e3b6f-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.823881 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.823700 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/989395fe-e528-4e51-96aa-e4b2a89e3b6f-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.823881 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.823845 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/989395fe-e528-4e51-96aa-e4b2a89e3b6f-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.823976 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.823902 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/989395fe-e528-4e51-96aa-e4b2a89e3b6f-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.824087 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.824070 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/989395fe-e528-4e51-96aa-e4b2a89e3b6f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.826258 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.826224 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/989395fe-e528-4e51-96aa-e4b2a89e3b6f-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.826532 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.826513 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/989395fe-e528-4e51-96aa-e4b2a89e3b6f-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.881602 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.881524 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/989395fe-e528-4e51-96aa-e4b2a89e3b6f-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.881745 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.881698 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7br6d\" (UniqueName: \"kubernetes.io/projected/989395fe-e528-4e51-96aa-e4b2a89e3b6f-kube-api-access-7br6d\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4\" (UID: \"989395fe-e528-4e51-96aa-e4b2a89e3b6f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:33.919530 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:33.919493 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:34.040237 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:34.040207 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj"] Oct 30 21:56:34.043067 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:56:34.043040 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb77193b8_0b50_4d53_beb9_619be2ea30b5.slice/crio-36ec34f1409e67be254250a407346c4827d79db422a337097068a07126c823c7 WatchSource:0}: Error finding container 36ec34f1409e67be254250a407346c4827d79db422a337097068a07126c823c7: Status 404 returned error can't find the container with id 36ec34f1409e67be254250a407346c4827d79db422a337097068a07126c823c7 Oct 30 21:56:34.044988 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:34.044972 6006 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:56:34.291769 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:34.291680 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4"] Oct 30 21:56:34.294012 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:56:34.293983 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989395fe_e528_4e51_96aa_e4b2a89e3b6f.slice/crio-6ed3d2d772d8644cd299a45ddcbcc9297e943b6b8fecb6164a899b799d817524 WatchSource:0}: Error finding container 6ed3d2d772d8644cd299a45ddcbcc9297e943b6b8fecb6164a899b799d817524: Status 404 returned error can't find the container with id 6ed3d2d772d8644cd299a45ddcbcc9297e943b6b8fecb6164a899b799d817524 Oct 30 21:56:34.338010 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:34.337974 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" event={"ID":"b77193b8-0b50-4d53-beb9-619be2ea30b5","Type":"ContainerStarted","Data":"36ec34f1409e67be254250a407346c4827d79db422a337097068a07126c823c7"} Oct 30 21:56:34.339130 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:34.339100 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" event={"ID":"989395fe-e528-4e51-96aa-e4b2a89e3b6f","Type":"ContainerStarted","Data":"6ed3d2d772d8644cd299a45ddcbcc9297e943b6b8fecb6164a899b799d817524"} Oct 30 21:56:38.231414 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:38.231367 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 21:56:38.231737 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:38.231448 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 21:56:38.231737 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:38.231485 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 21:56:38.364653 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:38.364568 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" event={"ID":"b77193b8-0b50-4d53-beb9-619be2ea30b5","Type":"ContainerStarted","Data":"1a4656ab03bb4e27ff95a8e1f968c1faaccb8aa82288c28504bc0dfb9d32e167"} Oct 30 21:56:38.454859 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:38.454801 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 21:56:38.454992 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:38.454918 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 21:56:38.454992 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:38.454968 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 21:56:38.553470 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:38.553417 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" podStartSLOduration=1.3680813760000001 podStartE2EDuration="5.553404251s" totalImagesPullingTime="4.185322875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:38.511849716 +0000 UTC m=+401.064026016" watchObservedRunningTime="2026-10-30 21:56:38.553404251 +0000 UTC m=+401.105580551" Oct 30 21:56:38.615790 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:38.615747 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:39.370440 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:39.370398 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" event={"ID":"989395fe-e528-4e51-96aa-e4b2a89e3b6f","Type":"ContainerStarted","Data":"aab6e6269d4fdd0a77a02fd0b1a2acacc9969765ff1d349aca7162098b27ba3a"} Oct 30 21:56:39.581879 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:39.581819 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" podStartSLOduration=2.423686047 podStartE2EDuration="6.581803313s" totalImagesPullingTime="4.158117266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:39.534061173 +0000 UTC m=+402.086237508" watchObservedRunningTime="2026-10-30 21:56:39.581803313 +0000 UTC m=+402.133979613" Oct 30 21:56:39.620738 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:39.620664 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:39.920472 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:39.920379 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:39.925226 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:39.925198 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:40.374223 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:40.374190 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:40.374757 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:40.374424 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:40.375159 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:40.375140 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 21:56:40.375245 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:56:40.375196 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4" Oct 30 21:56:57.908832 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:56:57.908787 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 21:57:14.319212 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.319167 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k"] Oct 30 21:57:14.330396 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.330368 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k" Oct 30 21:57:14.436609 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.436573 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 21:57:14.436802 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.436771 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 21:57:14.436802 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.436771 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-ng7kl\"" Oct 30 21:57:14.436932 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.436777 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 21:57:14.436932 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.436830 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 21:57:14.436932 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.436858 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 21:57:14.437811 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.437791 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k"] Oct 30 21:57:14.484992 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.484960 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6b08b8b1-4749-4599-9a45-f19be0099154-manifests\") pod \"opendatahub-operator-controller-manager-748948fb58-2st4k\" (UID: \"6b08b8b1-4749-4599-9a45-f19be0099154\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k" Oct 30 21:57:14.485164 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.485014 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6b08b8b1-4749-4599-9a45-f19be0099154-webhook-cert\") pod \"opendatahub-operator-controller-manager-748948fb58-2st4k\" (UID: \"6b08b8b1-4749-4599-9a45-f19be0099154\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k" Oct 30 21:57:14.485164 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.485127 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b08b8b1-4749-4599-9a45-f19be0099154-metrics-cert\") pod \"opendatahub-operator-controller-manager-748948fb58-2st4k\" (UID: \"6b08b8b1-4749-4599-9a45-f19be0099154\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k" Oct 30 21:57:14.485242 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.485161 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b08b8b1-4749-4599-9a45-f19be0099154-tmp\") pod \"opendatahub-operator-controller-manager-748948fb58-2st4k\" (UID: \"6b08b8b1-4749-4599-9a45-f19be0099154\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k" Oct 30 21:57:14.485242 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.485184 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6b08b8b1-4749-4599-9a45-f19be0099154-apiservice-cert\") pod \"opendatahub-operator-controller-manager-748948fb58-2st4k\" (UID: \"6b08b8b1-4749-4599-9a45-f19be0099154\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k" Oct 30 21:57:14.485352 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.485303 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xx2f\" (UniqueName: \"kubernetes.io/projected/6b08b8b1-4749-4599-9a45-f19be0099154-kube-api-access-4xx2f\") pod \"opendatahub-operator-controller-manager-748948fb58-2st4k\" (UID: \"6b08b8b1-4749-4599-9a45-f19be0099154\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k" Oct 30 21:57:14.586347 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.586231 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6b08b8b1-4749-4599-9a45-f19be0099154-webhook-cert\") pod \"opendatahub-operator-controller-manager-748948fb58-2st4k\" (UID: \"6b08b8b1-4749-4599-9a45-f19be0099154\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k" Oct 30 21:57:14.586347 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.586322 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b08b8b1-4749-4599-9a45-f19be0099154-metrics-cert\") pod \"opendatahub-operator-controller-manager-748948fb58-2st4k\" (UID: \"6b08b8b1-4749-4599-9a45-f19be0099154\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k" Oct 30 21:57:14.586347 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.586346 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b08b8b1-4749-4599-9a45-f19be0099154-tmp\") pod \"opendatahub-operator-controller-manager-748948fb58-2st4k\" (UID: \"6b08b8b1-4749-4599-9a45-f19be0099154\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k" Oct 30 21:57:14.586620 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.586367 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6b08b8b1-4749-4599-9a45-f19be0099154-apiservice-cert\") pod \"opendatahub-operator-controller-manager-748948fb58-2st4k\" (UID: \"6b08b8b1-4749-4599-9a45-f19be0099154\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k" Oct 30 21:57:14.586620 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.586390 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xx2f\" (UniqueName: \"kubernetes.io/projected/6b08b8b1-4749-4599-9a45-f19be0099154-kube-api-access-4xx2f\") pod \"opendatahub-operator-controller-manager-748948fb58-2st4k\" (UID: \"6b08b8b1-4749-4599-9a45-f19be0099154\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k" Oct 30 21:57:14.586620 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.586412 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6b08b8b1-4749-4599-9a45-f19be0099154-manifests\") pod \"opendatahub-operator-controller-manager-748948fb58-2st4k\" (UID: \"6b08b8b1-4749-4599-9a45-f19be0099154\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k" Oct 30 21:57:14.586872 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.586843 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6b08b8b1-4749-4599-9a45-f19be0099154-manifests\") pod \"opendatahub-operator-controller-manager-748948fb58-2st4k\" (UID: \"6b08b8b1-4749-4599-9a45-f19be0099154\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k" Oct 30 21:57:14.588744 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.588720 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b08b8b1-4749-4599-9a45-f19be0099154-tmp\") pod \"opendatahub-operator-controller-manager-748948fb58-2st4k\" (UID: \"6b08b8b1-4749-4599-9a45-f19be0099154\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k" Oct 30 21:57:14.588903 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.588876 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6b08b8b1-4749-4599-9a45-f19be0099154-apiservice-cert\") pod \"opendatahub-operator-controller-manager-748948fb58-2st4k\" (UID: \"6b08b8b1-4749-4599-9a45-f19be0099154\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k" Oct 30 21:57:14.589214 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.589197 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6b08b8b1-4749-4599-9a45-f19be0099154-webhook-cert\") pod \"opendatahub-operator-controller-manager-748948fb58-2st4k\" (UID: \"6b08b8b1-4749-4599-9a45-f19be0099154\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k" Oct 30 21:57:14.589351 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.589330 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b08b8b1-4749-4599-9a45-f19be0099154-metrics-cert\") pod \"opendatahub-operator-controller-manager-748948fb58-2st4k\" (UID: \"6b08b8b1-4749-4599-9a45-f19be0099154\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k" Oct 30 21:57:14.645542 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.645508 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xx2f\" (UniqueName: \"kubernetes.io/projected/6b08b8b1-4749-4599-9a45-f19be0099154-kube-api-access-4xx2f\") pod \"opendatahub-operator-controller-manager-748948fb58-2st4k\" (UID: \"6b08b8b1-4749-4599-9a45-f19be0099154\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k" Oct 30 21:57:14.647386 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:14.647359 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k" Oct 30 21:57:15.163521 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:15.163492 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k"] Oct 30 21:57:15.165746 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:57:15.165720 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b08b8b1_4749_4599_9a45_f19be0099154.slice/crio-a3d977bf694ab6ff4eb8602488bf7befc33bcc87b3d7b70b6d0c299e4cef419b WatchSource:0}: Error finding container a3d977bf694ab6ff4eb8602488bf7befc33bcc87b3d7b70b6d0c299e4cef419b: Status 404 returned error can't find the container with id a3d977bf694ab6ff4eb8602488bf7befc33bcc87b3d7b70b6d0c299e4cef419b Oct 30 21:57:15.520139 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:15.520046 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k" event={"ID":"6b08b8b1-4749-4599-9a45-f19be0099154","Type":"ContainerStarted","Data":"a3d977bf694ab6ff4eb8602488bf7befc33bcc87b3d7b70b6d0c299e4cef419b"} Oct 30 21:57:19.542302 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:19.542262 6006 generic.go:356] "Generic (PLEG): container finished" podID="6b08b8b1-4749-4599-9a45-f19be0099154" containerID="3d4c7b7fd9e32723ff53fcaa95210d75a3f3415b22151abf2d996d16d389b06b" exitCode=0 Oct 30 21:57:19.542722 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:19.542347 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k" event={"ID":"6b08b8b1-4749-4599-9a45-f19be0099154","Type":"ContainerDied","Data":"3d4c7b7fd9e32723ff53fcaa95210d75a3f3415b22151abf2d996d16d389b06b"} Oct 30 21:57:20.548963 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:20.548927 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k" event={"ID":"6b08b8b1-4749-4599-9a45-f19be0099154","Type":"ContainerStarted","Data":"c8eda76f2f4a777b39b2a8bc8d2b22053b23730989afabd87f2a8d256fcdbd5a"} Oct 30 21:57:20.549407 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:20.549106 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k" Oct 30 21:57:20.786305 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:20.786169 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k" podStartSLOduration=1.932234563 podStartE2EDuration="6.786154419s" totalImagesPullingTime="4.320989727s" totalInitContainerRuntime="532.930129ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:20.738566236 +0000 UTC m=+443.290742546" watchObservedRunningTime="2026-10-30 21:57:20.786154419 +0000 UTC m=+443.338330809" Oct 30 21:57:25.948798 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:25.948762 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5tr7h"] Oct 30 21:57:25.955486 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:25.955462 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5tr7h" Oct 30 21:57:25.985325 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:25.985282 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9465q\" (UniqueName: \"kubernetes.io/projected/55aa8d5b-9f21-423d-b736-34288116ba77-kube-api-access-9465q\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5tr7h\" (UID: \"55aa8d5b-9f21-423d-b736-34288116ba77\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5tr7h" Oct 30 21:57:25.985506 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:25.985448 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55aa8d5b-9f21-423d-b736-34288116ba77-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5tr7h\" (UID: \"55aa8d5b-9f21-423d-b736-34288116ba77\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5tr7h" Oct 30 21:57:26.004968 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:26.004933 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5tr7h"] Oct 30 21:57:26.050642 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:26.050607 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-84lq7\"" Oct 30 21:57:26.050642 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:26.050633 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 21:57:26.050859 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:26.050643 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 21:57:26.085987 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:26.085953 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55aa8d5b-9f21-423d-b736-34288116ba77-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5tr7h\" (UID: \"55aa8d5b-9f21-423d-b736-34288116ba77\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5tr7h" Oct 30 21:57:26.086142 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:26.086001 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9465q\" (UniqueName: \"kubernetes.io/projected/55aa8d5b-9f21-423d-b736-34288116ba77-kube-api-access-9465q\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5tr7h\" (UID: \"55aa8d5b-9f21-423d-b736-34288116ba77\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5tr7h" Oct 30 21:57:26.086404 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:26.086383 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55aa8d5b-9f21-423d-b736-34288116ba77-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5tr7h\" (UID: \"55aa8d5b-9f21-423d-b736-34288116ba77\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5tr7h" Oct 30 21:57:26.151832 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:26.151799 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9465q\" (UniqueName: \"kubernetes.io/projected/55aa8d5b-9f21-423d-b736-34288116ba77-kube-api-access-9465q\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5tr7h\" (UID: \"55aa8d5b-9f21-423d-b736-34288116ba77\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5tr7h" Oct 30 21:57:26.268295 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:26.268186 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5tr7h" Oct 30 21:57:26.761599 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:26.761561 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5tr7h"] Oct 30 21:57:26.771443 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:57:26.771405 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55aa8d5b_9f21_423d_b736_34288116ba77.slice/crio-e92a7f8b3a6fb654df5f1ba950150853cca075e5d85b22cf448f7249326d5a0c WatchSource:0}: Error finding container e92a7f8b3a6fb654df5f1ba950150853cca075e5d85b22cf448f7249326d5a0c: Status 404 returned error can't find the container with id e92a7f8b3a6fb654df5f1ba950150853cca075e5d85b22cf448f7249326d5a0c Oct 30 21:57:27.017550 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:27.017474 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zdnfl"] Oct 30 21:57:27.022337 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:27.022317 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zdnfl" Oct 30 21:57:27.074299 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:27.074237 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:57:27.074299 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:27.074238 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-fd2q5\"" Oct 30 21:57:27.074299 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:27.074295 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 21:57:27.096308 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:27.096277 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1957d64e-27f1-49e4-bc92-85dadc8253d0-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-zdnfl\" (UID: \"1957d64e-27f1-49e4-bc92-85dadc8253d0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zdnfl" Oct 30 21:57:27.096466 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:27.096416 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x77vd\" (UniqueName: \"kubernetes.io/projected/1957d64e-27f1-49e4-bc92-85dadc8253d0-kube-api-access-x77vd\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-zdnfl\" (UID: \"1957d64e-27f1-49e4-bc92-85dadc8253d0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zdnfl" Oct 30 21:57:27.122451 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:27.122405 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zdnfl"] Oct 30 21:57:27.197542 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:27.197502 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x77vd\" (UniqueName: \"kubernetes.io/projected/1957d64e-27f1-49e4-bc92-85dadc8253d0-kube-api-access-x77vd\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-zdnfl\" (UID: \"1957d64e-27f1-49e4-bc92-85dadc8253d0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zdnfl" Oct 30 21:57:27.197719 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:27.197573 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1957d64e-27f1-49e4-bc92-85dadc8253d0-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-zdnfl\" (UID: \"1957d64e-27f1-49e4-bc92-85dadc8253d0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zdnfl" Oct 30 21:57:27.198030 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:27.198011 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1957d64e-27f1-49e4-bc92-85dadc8253d0-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-zdnfl\" (UID: \"1957d64e-27f1-49e4-bc92-85dadc8253d0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zdnfl" Oct 30 21:57:27.256963 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:27.256927 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x77vd\" (UniqueName: \"kubernetes.io/projected/1957d64e-27f1-49e4-bc92-85dadc8253d0-kube-api-access-x77vd\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-zdnfl\" (UID: \"1957d64e-27f1-49e4-bc92-85dadc8253d0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zdnfl" Oct 30 21:57:27.335023 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:27.334981 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zdnfl" Oct 30 21:57:27.578432 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:27.578395 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5tr7h" event={"ID":"55aa8d5b-9f21-423d-b736-34288116ba77","Type":"ContainerStarted","Data":"e92a7f8b3a6fb654df5f1ba950150853cca075e5d85b22cf448f7249326d5a0c"} Oct 30 21:57:27.777456 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:27.777425 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zdnfl"] Oct 30 21:57:27.780045 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:57:27.780012 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1957d64e_27f1_49e4_bc92_85dadc8253d0.slice/crio-2213738b4a54b9eba25247a0ff6e9da4c7e1eb964d3ca3f1199bc1a7f2e3c73b WatchSource:0}: Error finding container 2213738b4a54b9eba25247a0ff6e9da4c7e1eb964d3ca3f1199bc1a7f2e3c73b: Status 404 returned error can't find the container with id 2213738b4a54b9eba25247a0ff6e9da4c7e1eb964d3ca3f1199bc1a7f2e3c73b Oct 30 21:57:27.909480 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:57:27.909443 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 21:57:28.585283 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:28.585222 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zdnfl" event={"ID":"1957d64e-27f1-49e4-bc92-85dadc8253d0","Type":"ContainerStarted","Data":"2213738b4a54b9eba25247a0ff6e9da4c7e1eb964d3ca3f1199bc1a7f2e3c73b"} Oct 30 21:57:31.555197 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:31.555150 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-748948fb58-2st4k" Oct 30 21:57:32.608435 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:32.608393 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5tr7h" event={"ID":"55aa8d5b-9f21-423d-b736-34288116ba77","Type":"ContainerStarted","Data":"4e0e83b671b77954d295b7fa4e96cea13841801d6cd61504d1253478236ed715"} Oct 30 21:57:32.608841 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:32.608487 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5tr7h" Oct 30 21:57:32.610410 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:32.610383 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zdnfl" event={"ID":"1957d64e-27f1-49e4-bc92-85dadc8253d0","Type":"ContainerStarted","Data":"02b29b88f05fc5038717c409d68534918b4d5280a36605f3f545b1ae128f55d4"} Oct 30 21:57:32.729116 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:32.729061 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5tr7h" podStartSLOduration=2.237536332 podStartE2EDuration="7.729049363s" totalImagesPullingTime="5.491513031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:32.72831574 +0000 UTC m=+455.280492042" watchObservedRunningTime="2026-10-30 21:57:32.729049363 +0000 UTC m=+455.281225663" Oct 30 21:57:32.843160 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:32.843102 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zdnfl" podStartSLOduration=2.633058995 podStartE2EDuration="6.843089624s" totalImagesPullingTime="4.210030629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:32.841833394 +0000 UTC m=+455.394009696" watchObservedRunningTime="2026-10-30 21:57:32.843089624 +0000 UTC m=+455.395265926" Oct 30 21:57:37.429874 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:37.429791 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-7gcxz"] Oct 30 21:57:37.433273 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:37.433237 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7gcxz" Oct 30 21:57:37.499826 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:37.499794 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flcwr\" (UniqueName: \"kubernetes.io/projected/e7333812-2624-427a-ac27-7f953e3d517e-kube-api-access-flcwr\") pod \"limitador-operator-controller-manager-5778bb4449-7gcxz\" (UID: \"e7333812-2624-427a-ac27-7f953e3d517e\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7gcxz" Oct 30 21:57:37.533359 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:37.533323 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tbxt2\"" Oct 30 21:57:37.534590 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:37.534567 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-7gcxz"] Oct 30 21:57:37.600353 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:37.600320 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flcwr\" (UniqueName: \"kubernetes.io/projected/e7333812-2624-427a-ac27-7f953e3d517e-kube-api-access-flcwr\") pod \"limitador-operator-controller-manager-5778bb4449-7gcxz\" (UID: \"e7333812-2624-427a-ac27-7f953e3d517e\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7gcxz" Oct 30 21:57:37.670048 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:37.670011 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flcwr\" (UniqueName: \"kubernetes.io/projected/e7333812-2624-427a-ac27-7f953e3d517e-kube-api-access-flcwr\") pod \"limitador-operator-controller-manager-5778bb4449-7gcxz\" (UID: \"e7333812-2624-427a-ac27-7f953e3d517e\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7gcxz" Oct 30 21:57:37.746840 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:37.746744 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7gcxz" Oct 30 21:57:38.113297 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:38.113117 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-7gcxz"] Oct 30 21:57:38.116055 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:57:38.116020 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7333812_2624_427a_ac27_7f953e3d517e.slice/crio-52306837b4a476784c4fe4ad04e1ccc542b77e3d85c6864a80733135839aeaf1 WatchSource:0}: Error finding container 52306837b4a476784c4fe4ad04e1ccc542b77e3d85c6864a80733135839aeaf1: Status 404 returned error can't find the container with id 52306837b4a476784c4fe4ad04e1ccc542b77e3d85c6864a80733135839aeaf1 Oct 30 21:57:38.635498 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:38.635459 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7gcxz" event={"ID":"e7333812-2624-427a-ac27-7f953e3d517e","Type":"ContainerStarted","Data":"52306837b4a476784c4fe4ad04e1ccc542b77e3d85c6864a80733135839aeaf1"} Oct 30 21:57:41.652130 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:41.652091 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7gcxz" event={"ID":"e7333812-2624-427a-ac27-7f953e3d517e","Type":"ContainerStarted","Data":"5b345d3c75dab706caf250f6f900d657482db7d02b11304ff8233e3a5bd0b36a"} Oct 30 21:57:41.652604 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:41.652244 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7gcxz" Oct 30 21:57:41.765295 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:41.765221 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7gcxz" podStartSLOduration=1.98620618 podStartE2EDuration="4.765205587s" totalImagesPullingTime="2.778999407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:41.763823049 +0000 UTC m=+464.315999350" watchObservedRunningTime="2026-10-30 21:57:41.765205587 +0000 UTC m=+464.317381887" Oct 30 21:57:43.616269 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:43.616212 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5tr7h" Oct 30 21:57:44.206514 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:44.206479 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4hvz4"] Oct 30 21:57:44.210042 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:44.210025 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4hvz4" Oct 30 21:57:44.263505 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:44.263470 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/297029e1-47c7-4aca-8699-4d94f72cd104-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4hvz4\" (UID: \"297029e1-47c7-4aca-8699-4d94f72cd104\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4hvz4" Oct 30 21:57:44.263698 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:44.263519 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ldjj\" (UniqueName: \"kubernetes.io/projected/297029e1-47c7-4aca-8699-4d94f72cd104-kube-api-access-4ldjj\") pod \"kuadrant-operator-controller-manager-678d87766f-4hvz4\" (UID: \"297029e1-47c7-4aca-8699-4d94f72cd104\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4hvz4" Oct 30 21:57:44.308721 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:44.308689 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4hvz4"] Oct 30 21:57:44.364955 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:44.364918 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/297029e1-47c7-4aca-8699-4d94f72cd104-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4hvz4\" (UID: \"297029e1-47c7-4aca-8699-4d94f72cd104\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4hvz4" Oct 30 21:57:44.365185 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:44.364963 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ldjj\" (UniqueName: \"kubernetes.io/projected/297029e1-47c7-4aca-8699-4d94f72cd104-kube-api-access-4ldjj\") pod \"kuadrant-operator-controller-manager-678d87766f-4hvz4\" (UID: \"297029e1-47c7-4aca-8699-4d94f72cd104\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4hvz4" Oct 30 21:57:44.365419 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:44.365397 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/297029e1-47c7-4aca-8699-4d94f72cd104-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4hvz4\" (UID: \"297029e1-47c7-4aca-8699-4d94f72cd104\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4hvz4" Oct 30 21:57:44.427140 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:44.427107 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ldjj\" (UniqueName: \"kubernetes.io/projected/297029e1-47c7-4aca-8699-4d94f72cd104-kube-api-access-4ldjj\") pod \"kuadrant-operator-controller-manager-678d87766f-4hvz4\" (UID: \"297029e1-47c7-4aca-8699-4d94f72cd104\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4hvz4" Oct 30 21:57:44.522714 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:44.522627 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4hvz4" Oct 30 21:57:44.938517 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:44.938476 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4hvz4"] Oct 30 21:57:44.941664 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:57:44.941634 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297029e1_47c7_4aca_8699_4d94f72cd104.slice/crio-9e48fb2dff6f4505ca6ea44d9485d2ecbc7b74a5c36b3dc0a1518ba504906100 WatchSource:0}: Error finding container 9e48fb2dff6f4505ca6ea44d9485d2ecbc7b74a5c36b3dc0a1518ba504906100: Status 404 returned error can't find the container with id 9e48fb2dff6f4505ca6ea44d9485d2ecbc7b74a5c36b3dc0a1518ba504906100 Oct 30 21:57:45.677086 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:45.677039 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4hvz4" event={"ID":"297029e1-47c7-4aca-8699-4d94f72cd104","Type":"ContainerStarted","Data":"6fbd3a5356dc52d067bf4726fc94d9c090ee4a5a6d489cce674c93c79f6bb0b9"} Oct 30 21:57:45.677397 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:45.677096 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4hvz4" event={"ID":"297029e1-47c7-4aca-8699-4d94f72cd104","Type":"ContainerStarted","Data":"9e48fb2dff6f4505ca6ea44d9485d2ecbc7b74a5c36b3dc0a1518ba504906100"} Oct 30 21:57:45.677397 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:45.677136 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4hvz4" Oct 30 21:57:45.815494 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:45.815443 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4hvz4" podStartSLOduration=1.815428932 podStartE2EDuration="1.815428932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:45.815022479 +0000 UTC m=+468.367198780" watchObservedRunningTime="2026-10-30 21:57:45.815428932 +0000 UTC m=+468.367605231" Oct 30 21:57:49.933228 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:49.933191 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft"] Oct 30 21:57:49.939515 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:49.939491 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:49.989338 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:49.989306 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-hhr5k\"" Oct 30 21:57:50.020062 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.020012 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e592cd54-dc62-473d-9993-976186532144-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.020278 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.020070 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e592cd54-dc62-473d-9993-976186532144-istio-token\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.020278 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.020210 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e592cd54-dc62-473d-9993-976186532144-istio-data\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.020422 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.020300 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e592cd54-dc62-473d-9993-976186532144-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.020422 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.020331 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e592cd54-dc62-473d-9993-976186532144-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.020422 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.020348 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e592cd54-dc62-473d-9993-976186532144-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.020590 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.020466 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx9xp\" (UniqueName: \"kubernetes.io/projected/e592cd54-dc62-473d-9993-976186532144-kube-api-access-rx9xp\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.020590 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.020497 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e592cd54-dc62-473d-9993-976186532144-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.020590 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.020521 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e592cd54-dc62-473d-9993-976186532144-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.037628 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.037590 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft"] Oct 30 21:57:50.121318 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.121282 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e592cd54-dc62-473d-9993-976186532144-istio-data\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.121522 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.121328 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e592cd54-dc62-473d-9993-976186532144-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.121522 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.121355 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e592cd54-dc62-473d-9993-976186532144-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.121522 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.121377 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e592cd54-dc62-473d-9993-976186532144-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.121704 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.121526 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx9xp\" (UniqueName: \"kubernetes.io/projected/e592cd54-dc62-473d-9993-976186532144-kube-api-access-rx9xp\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.121704 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.121586 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e592cd54-dc62-473d-9993-976186532144-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.121704 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.121627 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e592cd54-dc62-473d-9993-976186532144-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.121704 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.121672 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e592cd54-dc62-473d-9993-976186532144-istio-data\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.121704 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.121689 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e592cd54-dc62-473d-9993-976186532144-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.121964 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.121721 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e592cd54-dc62-473d-9993-976186532144-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.121964 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.121747 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e592cd54-dc62-473d-9993-976186532144-istio-token\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.122197 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.122175 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e592cd54-dc62-473d-9993-976186532144-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.122396 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.122376 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e592cd54-dc62-473d-9993-976186532144-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.122472 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.122413 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e592cd54-dc62-473d-9993-976186532144-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.123862 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.123832 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e592cd54-dc62-473d-9993-976186532144-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.124034 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.124017 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e592cd54-dc62-473d-9993-976186532144-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.181334 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.181294 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e592cd54-dc62-473d-9993-976186532144-istio-token\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.181491 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.181375 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx9xp\" (UniqueName: \"kubernetes.io/projected/e592cd54-dc62-473d-9993-976186532144-kube-api-access-rx9xp\") pod \"data-science-gateway-data-science-gateway-class-65844c896fp8nft\" (UID: \"e592cd54-dc62-473d-9993-976186532144\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.255089 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.254996 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:50.675313 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.675271 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft"] Oct 30 21:57:50.705956 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:57:50.705928 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode592cd54_dc62_473d_9993_976186532144.slice/crio-85eb94a6e4db935c3cf47c67fe5bf901d556e526364c2d1ab59f5366b96330a9 WatchSource:0}: Error finding container 85eb94a6e4db935c3cf47c67fe5bf901d556e526364c2d1ab59f5366b96330a9: Status 404 returned error can't find the container with id 85eb94a6e4db935c3cf47c67fe5bf901d556e526364c2d1ab59f5366b96330a9 Oct 30 21:57:50.708205 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.708166 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 21:57:50.708319 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.708242 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 21:57:50.708319 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:50.708293 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 21:57:51.709602 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:51.709561 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" event={"ID":"e592cd54-dc62-473d-9993-976186532144","Type":"ContainerStarted","Data":"db61e2343855d231b8f4219623782315ac27ffbb169289ac860d92abfde50a2c"} Oct 30 21:57:51.710080 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:51.709628 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" event={"ID":"e592cd54-dc62-473d-9993-976186532144","Type":"ContainerStarted","Data":"85eb94a6e4db935c3cf47c67fe5bf901d556e526364c2d1ab59f5366b96330a9"} Oct 30 21:57:51.874458 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:51.874406 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" podStartSLOduration=2.874388559 podStartE2EDuration="2.874388559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:51.873324655 +0000 UTC m=+474.425500955" watchObservedRunningTime="2026-10-30 21:57:51.874388559 +0000 UTC m=+474.426564858" Oct 30 21:57:52.256128 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:52.256084 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:52.261518 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:52.261492 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:52.659156 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:52.659125 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7gcxz" Oct 30 21:57:52.713853 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:52.713817 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:52.714782 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:52.714758 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fp8nft" Oct 30 21:57:56.684691 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:56.684657 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4hvz4" Oct 30 21:57:56.902611 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:56.902580 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5tr7h"] Oct 30 21:57:56.902866 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:56.902838 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5tr7h" podUID="55aa8d5b-9f21-423d-b736-34288116ba77" containerName="manager" containerID="cri-o://4e0e83b671b77954d295b7fa4e96cea13841801d6cd61504d1253478236ed715" gracePeriod=10 Oct 30 21:57:57.158521 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:57.158496 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5tr7h" Oct 30 21:57:57.288073 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:57.287989 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55aa8d5b-9f21-423d-b736-34288116ba77-kube-api-access-9465q\" (UniqueName: \"kubernetes.io/projected/55aa8d5b-9f21-423d-b736-34288116ba77-kube-api-access-9465q\") pod \"55aa8d5b-9f21-423d-b736-34288116ba77\" (UID: \"55aa8d5b-9f21-423d-b736-34288116ba77\") " Oct 30 21:57:57.288073 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:57.288047 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55aa8d5b-9f21-423d-b736-34288116ba77-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55aa8d5b-9f21-423d-b736-34288116ba77-extensions-socket-volume\") pod \"55aa8d5b-9f21-423d-b736-34288116ba77\" (UID: \"55aa8d5b-9f21-423d-b736-34288116ba77\") " Oct 30 21:57:57.288543 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:57.288520 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55aa8d5b-9f21-423d-b736-34288116ba77-extensions-socket-volume" pod "55aa8d5b-9f21-423d-b736-34288116ba77" (UID: "55aa8d5b-9f21-423d-b736-34288116ba77"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:57.290490 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:57.290461 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55aa8d5b-9f21-423d-b736-34288116ba77-kube-api-access-9465q" pod "55aa8d5b-9f21-423d-b736-34288116ba77" (UID: "55aa8d5b-9f21-423d-b736-34288116ba77"). InnerVolumeSpecName "kube-api-access-9465q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:57.388987 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:57.388929 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9465q\" (UniqueName: \"kubernetes.io/projected/55aa8d5b-9f21-423d-b736-34288116ba77-kube-api-access-9465q\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:57.388987 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:57.388969 6006 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55aa8d5b-9f21-423d-b736-34288116ba77-extensions-socket-volume\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:57.742681 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:57.742646 6006 generic.go:356] "Generic (PLEG): container finished" podID="55aa8d5b-9f21-423d-b736-34288116ba77" containerID="4e0e83b671b77954d295b7fa4e96cea13841801d6cd61504d1253478236ed715" exitCode=0 Oct 30 21:57:57.743092 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:57.742693 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5tr7h" Oct 30 21:57:57.743092 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:57.742707 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5tr7h" event={"ID":"55aa8d5b-9f21-423d-b736-34288116ba77","Type":"ContainerDied","Data":"4e0e83b671b77954d295b7fa4e96cea13841801d6cd61504d1253478236ed715"} Oct 30 21:57:57.743092 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:57.742750 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5tr7h" event={"ID":"55aa8d5b-9f21-423d-b736-34288116ba77","Type":"ContainerDied","Data":"e92a7f8b3a6fb654df5f1ba950150853cca075e5d85b22cf448f7249326d5a0c"} Oct 30 21:57:57.743092 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:57.742772 6006 scope.go:122] "RemoveContainer" containerID="4e0e83b671b77954d295b7fa4e96cea13841801d6cd61504d1253478236ed715" Oct 30 21:57:57.751323 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:57.751297 6006 scope.go:122] "RemoveContainer" containerID="4e0e83b671b77954d295b7fa4e96cea13841801d6cd61504d1253478236ed715" Oct 30 21:57:57.751609 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:57:57.751588 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e0e83b671b77954d295b7fa4e96cea13841801d6cd61504d1253478236ed715\": container with ID starting with 4e0e83b671b77954d295b7fa4e96cea13841801d6cd61504d1253478236ed715 not found: ID does not exist" containerID="4e0e83b671b77954d295b7fa4e96cea13841801d6cd61504d1253478236ed715" Oct 30 21:57:57.751678 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:57.751623 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e0e83b671b77954d295b7fa4e96cea13841801d6cd61504d1253478236ed715"} err="failed to get container status \"4e0e83b671b77954d295b7fa4e96cea13841801d6cd61504d1253478236ed715\": rpc error: code = NotFound desc = could not find container \"4e0e83b671b77954d295b7fa4e96cea13841801d6cd61504d1253478236ed715\": container with ID starting with 4e0e83b671b77954d295b7fa4e96cea13841801d6cd61504d1253478236ed715 not found: ID does not exist" Oct 30 21:57:57.912037 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:57:57.911937 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 21:57:57.929817 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:57.929782 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5tr7h"] Oct 30 21:57:57.976964 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:57.976933 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5tr7h"] Oct 30 21:57:59.243620 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:59.243582 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-sc4mf"] Oct 30 21:57:59.244050 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:59.244010 6006 cpu_manager.go:414] "containerMap: removing container" podUID="55aa8d5b-9f21-423d-b736-34288116ba77" containerName="manager" Oct 30 21:57:59.244050 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:59.244024 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55aa8d5b-9f21-423d-b736-34288116ba77" containerName="manager" Oct 30 21:57:59.244135 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:59.244102 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="55aa8d5b-9f21-423d-b736-34288116ba77" containerName="manager" Oct 30 21:57:59.248617 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:59.248598 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-sc4mf" Oct 30 21:57:59.300866 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:59.300828 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-sc4mf"] Oct 30 21:57:59.346510 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:59.346471 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-l5rh9\"" Oct 30 21:57:59.407017 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:59.406979 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7cnl\" (UniqueName: \"kubernetes.io/projected/3c069cf5-8c4c-46bb-85dd-5c6922869632-kube-api-access-w7cnl\") pod \"authorino-57f7f8659f-sc4mf\" (UID: \"3c069cf5-8c4c-46bb-85dd-5c6922869632\") " pod="kuadrant-system/authorino-57f7f8659f-sc4mf" Oct 30 21:57:59.508033 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:59.507936 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7cnl\" (UniqueName: \"kubernetes.io/projected/3c069cf5-8c4c-46bb-85dd-5c6922869632-kube-api-access-w7cnl\") pod \"authorino-57f7f8659f-sc4mf\" (UID: \"3c069cf5-8c4c-46bb-85dd-5c6922869632\") " pod="kuadrant-system/authorino-57f7f8659f-sc4mf" Oct 30 21:57:59.570343 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:59.570311 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7cnl\" (UniqueName: \"kubernetes.io/projected/3c069cf5-8c4c-46bb-85dd-5c6922869632-kube-api-access-w7cnl\") pod \"authorino-57f7f8659f-sc4mf\" (UID: \"3c069cf5-8c4c-46bb-85dd-5c6922869632\") " pod="kuadrant-system/authorino-57f7f8659f-sc4mf" Oct 30 21:57:59.602141 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:59.602105 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55aa8d5b-9f21-423d-b736-34288116ba77" path="/var/lib/kubelet/pods/55aa8d5b-9f21-423d-b736-34288116ba77/volumes" Oct 30 21:57:59.869900 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:57:59.869847 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-sc4mf" Oct 30 21:58:00.241683 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:00.241660 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-sc4mf"] Oct 30 21:58:00.244285 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:58:00.244237 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c069cf5_8c4c_46bb_85dd_5c6922869632.slice/crio-9eeeabeee3506209dee99d47c010ef4b92827d74c0a48e49b66d32deb9b373f9 WatchSource:0}: Error finding container 9eeeabeee3506209dee99d47c010ef4b92827d74c0a48e49b66d32deb9b373f9: Status 404 returned error can't find the container with id 9eeeabeee3506209dee99d47c010ef4b92827d74c0a48e49b66d32deb9b373f9 Oct 30 21:58:00.757887 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:00.757849 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-sc4mf" event={"ID":"3c069cf5-8c4c-46bb-85dd-5c6922869632","Type":"ContainerStarted","Data":"9eeeabeee3506209dee99d47c010ef4b92827d74c0a48e49b66d32deb9b373f9"} Oct 30 21:58:03.774425 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:03.774385 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-sc4mf" event={"ID":"3c069cf5-8c4c-46bb-85dd-5c6922869632","Type":"ContainerStarted","Data":"6bd94b3bc3b79ff56e1c5965c334b433dedbc0e45352f91bb021e654c218e93c"} Oct 30 21:58:03.883518 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:03.883460 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-sc4mf" podStartSLOduration=2.012976605 podStartE2EDuration="4.883444238s" totalImagesPullingTime="2.870467633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:03.882759267 +0000 UTC m=+486.434935570" watchObservedRunningTime="2026-10-30 21:58:03.883444238 +0000 UTC m=+486.435620557" Oct 30 21:58:17.161594 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:17.161553 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5lrcf"] Oct 30 21:58:17.165135 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:17.165112 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5lrcf" Oct 30 21:58:17.219626 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:17.219592 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5lrcf"] Oct 30 21:58:17.259796 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:17.259765 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpqh8\" (UniqueName: \"kubernetes.io/projected/c59f6fb6-03b1-44f0-82d6-536728dbb508-kube-api-access-hpqh8\") pod \"authorino-56c6c9ddd5-5lrcf\" (UID: \"c59f6fb6-03b1-44f0-82d6-536728dbb508\") " pod="kuadrant-system/authorino-56c6c9ddd5-5lrcf" Oct 30 21:58:17.360785 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:17.360746 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpqh8\" (UniqueName: \"kubernetes.io/projected/c59f6fb6-03b1-44f0-82d6-536728dbb508-kube-api-access-hpqh8\") pod \"authorino-56c6c9ddd5-5lrcf\" (UID: \"c59f6fb6-03b1-44f0-82d6-536728dbb508\") " pod="kuadrant-system/authorino-56c6c9ddd5-5lrcf" Oct 30 21:58:17.419553 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:17.419480 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpqh8\" (UniqueName: \"kubernetes.io/projected/c59f6fb6-03b1-44f0-82d6-536728dbb508-kube-api-access-hpqh8\") pod \"authorino-56c6c9ddd5-5lrcf\" (UID: \"c59f6fb6-03b1-44f0-82d6-536728dbb508\") " pod="kuadrant-system/authorino-56c6c9ddd5-5lrcf" Oct 30 21:58:17.476608 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:17.476567 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5lrcf" Oct 30 21:58:17.838592 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:17.838566 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5lrcf"] Oct 30 21:58:17.841473 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:58:17.841439 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc59f6fb6_03b1_44f0_82d6_536728dbb508.slice/crio-d84d4bc895448b52d62c44b9817db851c1f7d8e46bcf36381666a77129caaeb0 WatchSource:0}: Error finding container d84d4bc895448b52d62c44b9817db851c1f7d8e46bcf36381666a77129caaeb0: Status 404 returned error can't find the container with id d84d4bc895448b52d62c44b9817db851c1f7d8e46bcf36381666a77129caaeb0 Oct 30 21:58:18.197061 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:18.196963 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-xzv68"] Oct 30 21:58:18.201792 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:18.201766 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xzv68" Oct 30 21:58:18.251402 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:18.251368 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-h2cbw\"" Oct 30 21:58:18.268955 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:18.268922 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/15211cfe-5655-448d-b764-ae4de94fecb8-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-xzv68\" (UID: \"15211cfe-5655-448d-b764-ae4de94fecb8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xzv68" Oct 30 21:58:18.269124 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:18.268972 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/15211cfe-5655-448d-b764-ae4de94fecb8-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-xzv68\" (UID: \"15211cfe-5655-448d-b764-ae4de94fecb8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xzv68" Oct 30 21:58:18.269124 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:18.269065 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15211cfe-5655-448d-b764-ae4de94fecb8-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-xzv68\" (UID: \"15211cfe-5655-448d-b764-ae4de94fecb8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xzv68" Oct 30 21:58:18.269203 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:18.269183 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sdc7\" (UniqueName: \"kubernetes.io/projected/15211cfe-5655-448d-b764-ae4de94fecb8-kube-api-access-8sdc7\") pod \"kserve-module-controller-manager-7566cf5fc8-xzv68\" (UID: \"15211cfe-5655-448d-b764-ae4de94fecb8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xzv68" Oct 30 21:58:18.298397 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:18.298360 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 21:58:18.298631 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:18.298504 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 21:58:18.370706 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:18.370665 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sdc7\" (UniqueName: \"kubernetes.io/projected/15211cfe-5655-448d-b764-ae4de94fecb8-kube-api-access-8sdc7\") pod \"kserve-module-controller-manager-7566cf5fc8-xzv68\" (UID: \"15211cfe-5655-448d-b764-ae4de94fecb8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xzv68" Oct 30 21:58:18.370900 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:18.370722 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/15211cfe-5655-448d-b764-ae4de94fecb8-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-xzv68\" (UID: \"15211cfe-5655-448d-b764-ae4de94fecb8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xzv68" Oct 30 21:58:18.370900 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:18.370769 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/15211cfe-5655-448d-b764-ae4de94fecb8-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-xzv68\" (UID: \"15211cfe-5655-448d-b764-ae4de94fecb8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xzv68" Oct 30 21:58:18.370900 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:18.370801 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15211cfe-5655-448d-b764-ae4de94fecb8-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-xzv68\" (UID: \"15211cfe-5655-448d-b764-ae4de94fecb8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xzv68" Oct 30 21:58:18.371230 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:18.371206 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/15211cfe-5655-448d-b764-ae4de94fecb8-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-xzv68\" (UID: \"15211cfe-5655-448d-b764-ae4de94fecb8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xzv68" Oct 30 21:58:18.371302 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:18.371208 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15211cfe-5655-448d-b764-ae4de94fecb8-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-xzv68\" (UID: \"15211cfe-5655-448d-b764-ae4de94fecb8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xzv68" Oct 30 21:58:18.392902 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:18.392872 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 21:58:18.394393 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:18.394368 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-xzv68"] Oct 30 21:58:18.402174 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:18.402149 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/15211cfe-5655-448d-b764-ae4de94fecb8-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-xzv68\" (UID: \"15211cfe-5655-448d-b764-ae4de94fecb8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xzv68" Oct 30 21:58:18.429503 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:18.429471 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sdc7\" (UniqueName: \"kubernetes.io/projected/15211cfe-5655-448d-b764-ae4de94fecb8-kube-api-access-8sdc7\") pod \"kserve-module-controller-manager-7566cf5fc8-xzv68\" (UID: \"15211cfe-5655-448d-b764-ae4de94fecb8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xzv68" Oct 30 21:58:18.514793 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:18.514703 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xzv68" Oct 30 21:58:18.846000 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:18.845962 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5lrcf" event={"ID":"c59f6fb6-03b1-44f0-82d6-536728dbb508","Type":"ContainerStarted","Data":"d84d4bc895448b52d62c44b9817db851c1f7d8e46bcf36381666a77129caaeb0"} Oct 30 21:58:18.931397 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:18.931343 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-xzv68"] Oct 30 21:58:18.933727 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:58:18.933698 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15211cfe_5655_448d_b764_ae4de94fecb8.slice/crio-b04c7d2aa05a20218d40ab896644a92839989fd92fc84b52d757c7f68d0da931 WatchSource:0}: Error finding container b04c7d2aa05a20218d40ab896644a92839989fd92fc84b52d757c7f68d0da931: Status 404 returned error can't find the container with id b04c7d2aa05a20218d40ab896644a92839989fd92fc84b52d757c7f68d0da931 Oct 30 21:58:19.852007 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:19.851958 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xzv68" event={"ID":"15211cfe-5655-448d-b764-ae4de94fecb8","Type":"ContainerStarted","Data":"b04c7d2aa05a20218d40ab896644a92839989fd92fc84b52d757c7f68d0da931"} Oct 30 21:58:19.853503 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:19.853475 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5lrcf" event={"ID":"c59f6fb6-03b1-44f0-82d6-536728dbb508","Type":"ContainerStarted","Data":"015984b676650b628a76693da06fd13e228f0901ab7991e7dadb27a4b1bcab6a"} Oct 30 21:58:19.992067 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:19.992011 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-5lrcf" podStartSLOduration=1.539363384 podStartE2EDuration="2.99198324s" totalImagesPullingTime="1.452619856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:19.963625928 +0000 UTC m=+502.515802230" watchObservedRunningTime="2026-10-30 21:58:19.99198324 +0000 UTC m=+502.544159539" Oct 30 21:58:19.993100 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:19.993065 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-sc4mf"] Oct 30 21:58:19.993473 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:19.993446 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-sc4mf" podUID="3c069cf5-8c4c-46bb-85dd-5c6922869632" containerName="authorino" containerID="cri-o://6bd94b3bc3b79ff56e1c5965c334b433dedbc0e45352f91bb021e654c218e93c" gracePeriod=30 Oct 30 21:58:20.273847 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.273818 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-sc4mf" Oct 30 21:58:20.392891 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.392767 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c069cf5-8c4c-46bb-85dd-5c6922869632-kube-api-access-w7cnl\" (UniqueName: \"kubernetes.io/projected/3c069cf5-8c4c-46bb-85dd-5c6922869632-kube-api-access-w7cnl\") pod \"3c069cf5-8c4c-46bb-85dd-5c6922869632\" (UID: \"3c069cf5-8c4c-46bb-85dd-5c6922869632\") " Oct 30 21:58:20.393974 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.393934 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 21:58:20.394916 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.394524 6006 cpu_manager.go:414] "containerMap: removing container" podUID="3c069cf5-8c4c-46bb-85dd-5c6922869632" containerName="authorino" Oct 30 21:58:20.394916 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.394546 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c069cf5-8c4c-46bb-85dd-5c6922869632" containerName="authorino" Oct 30 21:58:20.394916 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.394647 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c069cf5-8c4c-46bb-85dd-5c6922869632" containerName="authorino" Oct 30 21:58:20.398379 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.398265 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-p76jm" Oct 30 21:58:20.398498 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.398404 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c069cf5-8c4c-46bb-85dd-5c6922869632-kube-api-access-w7cnl" pod "3c069cf5-8c4c-46bb-85dd-5c6922869632" (UID: "3c069cf5-8c4c-46bb-85dd-5c6922869632"). InnerVolumeSpecName "kube-api-access-w7cnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:20.449192 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.449151 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 21:58:20.453526 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.453486 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7m7zw\"" Oct 30 21:58:20.455184 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.455159 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 21:58:20.494179 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.494144 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4vdr\" (UniqueName: \"kubernetes.io/projected/47477b2f-5e3f-437c-8203-80aac978e56a-kube-api-access-g4vdr\") pod \"limitador-limitador-7c9b87d574-p76jm\" (UID: \"47477b2f-5e3f-437c-8203-80aac978e56a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-p76jm" Oct 30 21:58:20.494425 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.494233 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/47477b2f-5e3f-437c-8203-80aac978e56a-config-file\") pod \"limitador-limitador-7c9b87d574-p76jm\" (UID: \"47477b2f-5e3f-437c-8203-80aac978e56a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-p76jm" Oct 30 21:58:20.494425 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.494329 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7cnl\" (UniqueName: \"kubernetes.io/projected/3c069cf5-8c4c-46bb-85dd-5c6922869632-kube-api-access-w7cnl\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:20.595750 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.595526 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/47477b2f-5e3f-437c-8203-80aac978e56a-config-file\") pod \"limitador-limitador-7c9b87d574-p76jm\" (UID: \"47477b2f-5e3f-437c-8203-80aac978e56a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-p76jm" Oct 30 21:58:20.595920 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.595868 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4vdr\" (UniqueName: \"kubernetes.io/projected/47477b2f-5e3f-437c-8203-80aac978e56a-kube-api-access-g4vdr\") pod \"limitador-limitador-7c9b87d574-p76jm\" (UID: \"47477b2f-5e3f-437c-8203-80aac978e56a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-p76jm" Oct 30 21:58:20.596326 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.596242 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/47477b2f-5e3f-437c-8203-80aac978e56a-config-file\") pod \"limitador-limitador-7c9b87d574-p76jm\" (UID: \"47477b2f-5e3f-437c-8203-80aac978e56a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-p76jm" Oct 30 21:58:20.655429 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.655347 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4vdr\" (UniqueName: \"kubernetes.io/projected/47477b2f-5e3f-437c-8203-80aac978e56a-kube-api-access-g4vdr\") pod \"limitador-limitador-7c9b87d574-p76jm\" (UID: \"47477b2f-5e3f-437c-8203-80aac978e56a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-p76jm" Oct 30 21:58:20.680949 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.680913 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 21:58:20.724377 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.724340 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-p76jm" Oct 30 21:58:20.858949 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.858909 6006 generic.go:356] "Generic (PLEG): container finished" podID="3c069cf5-8c4c-46bb-85dd-5c6922869632" containerID="6bd94b3bc3b79ff56e1c5965c334b433dedbc0e45352f91bb021e654c218e93c" exitCode=0 Oct 30 21:58:20.858949 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.858948 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-sc4mf" Oct 30 21:58:20.859453 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.858996 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-sc4mf" event={"ID":"3c069cf5-8c4c-46bb-85dd-5c6922869632","Type":"ContainerDied","Data":"6bd94b3bc3b79ff56e1c5965c334b433dedbc0e45352f91bb021e654c218e93c"} Oct 30 21:58:20.859453 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.859040 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-sc4mf" event={"ID":"3c069cf5-8c4c-46bb-85dd-5c6922869632","Type":"ContainerDied","Data":"9eeeabeee3506209dee99d47c010ef4b92827d74c0a48e49b66d32deb9b373f9"} Oct 30 21:58:20.859453 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.859058 6006 scope.go:122] "RemoveContainer" containerID="6bd94b3bc3b79ff56e1c5965c334b433dedbc0e45352f91bb021e654c218e93c" Oct 30 21:58:20.868484 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.868461 6006 scope.go:122] "RemoveContainer" containerID="6bd94b3bc3b79ff56e1c5965c334b433dedbc0e45352f91bb021e654c218e93c" Oct 30 21:58:20.868745 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:58:20.868724 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bd94b3bc3b79ff56e1c5965c334b433dedbc0e45352f91bb021e654c218e93c\": container with ID starting with 6bd94b3bc3b79ff56e1c5965c334b433dedbc0e45352f91bb021e654c218e93c not found: ID does not exist" containerID="6bd94b3bc3b79ff56e1c5965c334b433dedbc0e45352f91bb021e654c218e93c" Oct 30 21:58:20.868806 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:20.868754 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bd94b3bc3b79ff56e1c5965c334b433dedbc0e45352f91bb021e654c218e93c"} err="failed to get container status \"6bd94b3bc3b79ff56e1c5965c334b433dedbc0e45352f91bb021e654c218e93c\": rpc error: code = NotFound desc = could not find container \"6bd94b3bc3b79ff56e1c5965c334b433dedbc0e45352f91bb021e654c218e93c\": container with ID starting with 6bd94b3bc3b79ff56e1c5965c334b433dedbc0e45352f91bb021e654c218e93c not found: ID does not exist" Oct 30 21:58:21.015203 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:21.015116 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-sc4mf"] Oct 30 21:58:21.019171 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:21.019145 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-sc4mf"] Oct 30 21:58:21.120971 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:21.120538 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 21:58:21.123851 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:58:21.123818 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47477b2f_5e3f_437c_8203_80aac978e56a.slice/crio-557e07de2b090796db6918dd2fc29a0d2016648e92ff00935b26c33065e45118 WatchSource:0}: Error finding container 557e07de2b090796db6918dd2fc29a0d2016648e92ff00935b26c33065e45118: Status 404 returned error can't find the container with id 557e07de2b090796db6918dd2fc29a0d2016648e92ff00935b26c33065e45118 Oct 30 21:58:21.609274 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:21.609210 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c069cf5-8c4c-46bb-85dd-5c6922869632" path="/var/lib/kubelet/pods/3c069cf5-8c4c-46bb-85dd-5c6922869632/volumes" Oct 30 21:58:21.868447 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:21.868346 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-p76jm" event={"ID":"47477b2f-5e3f-437c-8203-80aac978e56a","Type":"ContainerStarted","Data":"557e07de2b090796db6918dd2fc29a0d2016648e92ff00935b26c33065e45118"} Oct 30 21:58:22.874420 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:22.874380 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xzv68" event={"ID":"15211cfe-5655-448d-b764-ae4de94fecb8","Type":"ContainerStarted","Data":"63096599a84a2436f1f7782075aaaf2459e3e4fb57bdfab14fa670fa02724426"} Oct 30 21:58:22.874812 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:22.874446 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xzv68" Oct 30 21:58:23.036917 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:23.036857 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xzv68" podStartSLOduration=1.840708717 podStartE2EDuration="5.036844197s" totalImagesPullingTime="3.19613548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:23.036422737 +0000 UTC m=+505.588599036" watchObservedRunningTime="2026-10-30 21:58:23.036844197 +0000 UTC m=+505.589020497" Oct 30 21:58:25.890511 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:25.890469 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-p76jm" event={"ID":"47477b2f-5e3f-437c-8203-80aac978e56a","Type":"ContainerStarted","Data":"65edfd9d9d399f3710f4e03e4e4d5d89ad9aede48b5b1bb09917df503a1981f9"} Oct 30 21:58:25.890924 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:25.890561 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-p76jm" Oct 30 21:58:26.005530 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:26.005471 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-p76jm" podStartSLOduration=2.1661938259999998 podStartE2EDuration="6.005455515s" totalImagesPullingTime="3.839261689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:26.003507122 +0000 UTC m=+508.555683421" watchObservedRunningTime="2026-10-30 21:58:26.005455515 +0000 UTC m=+508.557631814" Oct 30 21:58:27.910823 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:58:27.910785 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 21:58:36.897340 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:36.897308 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-p76jm" Oct 30 21:58:43.880113 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:58:43.880078 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xzv68" Oct 30 21:58:57.910645 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:58:57.910501 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 21:59:25.466608 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.466524 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-68ns6"] Oct 30 21:59:25.473696 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.473670 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-68ns6" Oct 30 21:59:25.527960 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.527914 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-68ns6"] Oct 30 21:59:25.571812 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.571758 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-zhrdc\"" Oct 30 21:59:25.571988 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.571770 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 21:59:25.573255 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.573224 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-wb5v2"] Oct 30 21:59:25.576947 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.576931 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wb5v2" Oct 30 21:59:25.617607 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.617566 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfpr6\" (UniqueName: \"kubernetes.io/projected/f001662b-cac0-4ac1-a34f-d6bc732ee69f-kube-api-access-sfpr6\") pod \"kserve-controller-manager-776cc66d47-68ns6\" (UID: \"f001662b-cac0-4ac1-a34f-d6bc732ee69f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-68ns6" Oct 30 21:59:25.617779 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.617732 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f001662b-cac0-4ac1-a34f-d6bc732ee69f-cert\") pod \"kserve-controller-manager-776cc66d47-68ns6\" (UID: \"f001662b-cac0-4ac1-a34f-d6bc732ee69f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-68ns6" Oct 30 21:59:25.628133 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.628101 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-wb5v2"] Oct 30 21:59:25.638336 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.638306 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-gnw9z\"" Oct 30 21:59:25.638491 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.638342 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 21:59:25.718665 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.718559 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f001662b-cac0-4ac1-a34f-d6bc732ee69f-cert\") pod \"kserve-controller-manager-776cc66d47-68ns6\" (UID: \"f001662b-cac0-4ac1-a34f-d6bc732ee69f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-68ns6" Oct 30 21:59:25.718665 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.718619 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk7d2\" (UniqueName: \"kubernetes.io/projected/48d0218e-59fe-4f51-9b47-df84046b0486-kube-api-access-wk7d2\") pod \"llmisvc-controller-manager-869d5dcc5d-wb5v2\" (UID: \"48d0218e-59fe-4f51-9b47-df84046b0486\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wb5v2" Oct 30 21:59:25.718911 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.718672 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48d0218e-59fe-4f51-9b47-df84046b0486-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-wb5v2\" (UID: \"48d0218e-59fe-4f51-9b47-df84046b0486\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wb5v2" Oct 30 21:59:25.718911 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.718751 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfpr6\" (UniqueName: \"kubernetes.io/projected/f001662b-cac0-4ac1-a34f-d6bc732ee69f-kube-api-access-sfpr6\") pod \"kserve-controller-manager-776cc66d47-68ns6\" (UID: \"f001662b-cac0-4ac1-a34f-d6bc732ee69f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-68ns6" Oct 30 21:59:25.721346 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.721320 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f001662b-cac0-4ac1-a34f-d6bc732ee69f-cert\") pod \"kserve-controller-manager-776cc66d47-68ns6\" (UID: \"f001662b-cac0-4ac1-a34f-d6bc732ee69f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-68ns6" Oct 30 21:59:25.778024 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.777988 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfpr6\" (UniqueName: \"kubernetes.io/projected/f001662b-cac0-4ac1-a34f-d6bc732ee69f-kube-api-access-sfpr6\") pod \"kserve-controller-manager-776cc66d47-68ns6\" (UID: \"f001662b-cac0-4ac1-a34f-d6bc732ee69f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-68ns6" Oct 30 21:59:25.786845 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.786812 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-68ns6" Oct 30 21:59:25.794180 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.794147 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-bg2f2"] Oct 30 21:59:25.797958 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.797936 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-bg2f2" Oct 30 21:59:25.820345 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.820308 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk7d2\" (UniqueName: \"kubernetes.io/projected/48d0218e-59fe-4f51-9b47-df84046b0486-kube-api-access-wk7d2\") pod \"llmisvc-controller-manager-869d5dcc5d-wb5v2\" (UID: \"48d0218e-59fe-4f51-9b47-df84046b0486\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wb5v2" Oct 30 21:59:25.820491 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.820370 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48d0218e-59fe-4f51-9b47-df84046b0486-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-wb5v2\" (UID: \"48d0218e-59fe-4f51-9b47-df84046b0486\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wb5v2" Oct 30 21:59:25.823096 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.823065 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48d0218e-59fe-4f51-9b47-df84046b0486-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-wb5v2\" (UID: \"48d0218e-59fe-4f51-9b47-df84046b0486\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wb5v2" Oct 30 21:59:25.850200 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.847789 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-64cff\"" Oct 30 21:59:25.850200 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.847864 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 21:59:25.850200 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.847986 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 21:59:25.854838 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.854808 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-bg2f2"] Oct 30 21:59:25.880366 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.880332 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk7d2\" (UniqueName: \"kubernetes.io/projected/48d0218e-59fe-4f51-9b47-df84046b0486-kube-api-access-wk7d2\") pod \"llmisvc-controller-manager-869d5dcc5d-wb5v2\" (UID: \"48d0218e-59fe-4f51-9b47-df84046b0486\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wb5v2" Oct 30 21:59:25.889141 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.889105 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wb5v2" Oct 30 21:59:25.921936 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.921894 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x8m4\" (UniqueName: \"kubernetes.io/projected/d46d3bef-fb7d-42c2-82ed-61a009a2c713-kube-api-access-8x8m4\") pod \"odh-model-controller-769875db56-bg2f2\" (UID: \"d46d3bef-fb7d-42c2-82ed-61a009a2c713\") " pod="opendatahub/odh-model-controller-769875db56-bg2f2" Oct 30 21:59:25.922093 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.922002 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d46d3bef-fb7d-42c2-82ed-61a009a2c713-cert\") pod \"odh-model-controller-769875db56-bg2f2\" (UID: \"d46d3bef-fb7d-42c2-82ed-61a009a2c713\") " pod="opendatahub/odh-model-controller-769875db56-bg2f2" Oct 30 21:59:25.922093 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:25.922087 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d46d3bef-fb7d-42c2-82ed-61a009a2c713-metrics-certs\") pod \"odh-model-controller-769875db56-bg2f2\" (UID: \"d46d3bef-fb7d-42c2-82ed-61a009a2c713\") " pod="opendatahub/odh-model-controller-769875db56-bg2f2" Oct 30 21:59:26.022696 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:26.022608 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x8m4\" (UniqueName: \"kubernetes.io/projected/d46d3bef-fb7d-42c2-82ed-61a009a2c713-kube-api-access-8x8m4\") pod \"odh-model-controller-769875db56-bg2f2\" (UID: \"d46d3bef-fb7d-42c2-82ed-61a009a2c713\") " pod="opendatahub/odh-model-controller-769875db56-bg2f2" Oct 30 21:59:26.022696 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:26.022652 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d46d3bef-fb7d-42c2-82ed-61a009a2c713-cert\") pod \"odh-model-controller-769875db56-bg2f2\" (UID: \"d46d3bef-fb7d-42c2-82ed-61a009a2c713\") " pod="opendatahub/odh-model-controller-769875db56-bg2f2" Oct 30 21:59:26.022881 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:26.022702 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d46d3bef-fb7d-42c2-82ed-61a009a2c713-metrics-certs\") pod \"odh-model-controller-769875db56-bg2f2\" (UID: \"d46d3bef-fb7d-42c2-82ed-61a009a2c713\") " pod="opendatahub/odh-model-controller-769875db56-bg2f2" Oct 30 21:59:26.025317 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:26.025296 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d46d3bef-fb7d-42c2-82ed-61a009a2c713-metrics-certs\") pod \"odh-model-controller-769875db56-bg2f2\" (UID: \"d46d3bef-fb7d-42c2-82ed-61a009a2c713\") " pod="opendatahub/odh-model-controller-769875db56-bg2f2" Oct 30 21:59:26.025374 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:26.025328 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d46d3bef-fb7d-42c2-82ed-61a009a2c713-cert\") pod \"odh-model-controller-769875db56-bg2f2\" (UID: \"d46d3bef-fb7d-42c2-82ed-61a009a2c713\") " pod="opendatahub/odh-model-controller-769875db56-bg2f2" Oct 30 21:59:26.082940 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:26.082907 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x8m4\" (UniqueName: \"kubernetes.io/projected/d46d3bef-fb7d-42c2-82ed-61a009a2c713-kube-api-access-8x8m4\") pod \"odh-model-controller-769875db56-bg2f2\" (UID: \"d46d3bef-fb7d-42c2-82ed-61a009a2c713\") " pod="opendatahub/odh-model-controller-769875db56-bg2f2" Oct 30 21:59:26.124137 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:26.124100 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-bg2f2" Oct 30 21:59:26.160225 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:26.160189 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-68ns6"] Oct 30 21:59:26.162236 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:59:26.162206 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf001662b_cac0_4ac1_a34f_d6bc732ee69f.slice/crio-d85502c2c19dfc2971abe9c1255f50bbb5aaade8ae6901d3c86921f00a43909e WatchSource:0}: Error finding container d85502c2c19dfc2971abe9c1255f50bbb5aaade8ae6901d3c86921f00a43909e: Status 404 returned error can't find the container with id d85502c2c19dfc2971abe9c1255f50bbb5aaade8ae6901d3c86921f00a43909e Oct 30 21:59:26.186063 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:26.186028 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-68ns6" event={"ID":"f001662b-cac0-4ac1-a34f-d6bc732ee69f","Type":"ContainerStarted","Data":"d85502c2c19dfc2971abe9c1255f50bbb5aaade8ae6901d3c86921f00a43909e"} Oct 30 21:59:26.257791 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:26.257767 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-wb5v2"] Oct 30 21:59:26.259958 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:59:26.259927 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod48d0218e_59fe_4f51_9b47_df84046b0486.slice/crio-e9719d856ea0d71ed6d1e83e391f2ea6ef3fafbe3e6000b643d06518b500a38b WatchSource:0}: Error finding container e9719d856ea0d71ed6d1e83e391f2ea6ef3fafbe3e6000b643d06518b500a38b: Status 404 returned error can't find the container with id e9719d856ea0d71ed6d1e83e391f2ea6ef3fafbe3e6000b643d06518b500a38b Oct 30 21:59:26.495500 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:26.495465 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-bg2f2"] Oct 30 21:59:26.499472 ip-10-0-135-7 kubenswrapper[6006]: W1030 21:59:26.499445 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd46d3bef_fb7d_42c2_82ed_61a009a2c713.slice/crio-71907f139783227e36c49e9d29b7357daf3c8308b9a8151220bcd85de250785a WatchSource:0}: Error finding container 71907f139783227e36c49e9d29b7357daf3c8308b9a8151220bcd85de250785a: Status 404 returned error can't find the container with id 71907f139783227e36c49e9d29b7357daf3c8308b9a8151220bcd85de250785a Oct 30 21:59:27.193987 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:27.193948 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-bg2f2" event={"ID":"d46d3bef-fb7d-42c2-82ed-61a009a2c713","Type":"ContainerStarted","Data":"71907f139783227e36c49e9d29b7357daf3c8308b9a8151220bcd85de250785a"} Oct 30 21:59:27.195511 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:27.195478 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wb5v2" event={"ID":"48d0218e-59fe-4f51-9b47-df84046b0486","Type":"ContainerStarted","Data":"e9719d856ea0d71ed6d1e83e391f2ea6ef3fafbe3e6000b643d06518b500a38b"} Oct 30 21:59:27.922984 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:59:27.922946 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 21:59:30.214425 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:30.214383 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-68ns6" event={"ID":"f001662b-cac0-4ac1-a34f-d6bc732ee69f","Type":"ContainerStarted","Data":"c0cb8955c8e946411552f3a247f22ddc47bd61a0a9bf98692fe32c31080f3673"} Oct 30 21:59:30.214876 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:30.214557 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-68ns6" Oct 30 21:59:30.216058 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:30.216023 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-bg2f2" event={"ID":"d46d3bef-fb7d-42c2-82ed-61a009a2c713","Type":"ContainerStarted","Data":"a47f737d3b7aed6a83fb2f58e36fb613a6d7294fa6685e6816175127c263956c"} Oct 30 21:59:30.216188 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:30.216168 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-bg2f2" Oct 30 21:59:30.217646 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:30.217619 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wb5v2" event={"ID":"48d0218e-59fe-4f51-9b47-df84046b0486","Type":"ContainerStarted","Data":"38d0d90ec30518b5d084071a90c909f7291af7011310ca0c104a7ea3ed5c4c5c"} Oct 30 21:59:30.217779 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:30.217761 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wb5v2" Oct 30 21:59:30.326149 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:30.326089 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-68ns6" podStartSLOduration=2.274687978 podStartE2EDuration="5.3260761s" totalImagesPullingTime="3.051388122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:30.325860726 +0000 UTC m=+572.878037027" watchObservedRunningTime="2026-10-30 21:59:30.3260761 +0000 UTC m=+572.878252394" Oct 30 21:59:30.438998 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:30.438871 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wb5v2" podStartSLOduration=2.275112326 podStartE2EDuration="5.438850542s" totalImagesPullingTime="3.163738216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:30.43748527 +0000 UTC m=+572.989661569" watchObservedRunningTime="2026-10-30 21:59:30.438850542 +0000 UTC m=+572.991026846" Oct 30 21:59:30.551489 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:30.551431 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-bg2f2" podStartSLOduration=1.9101341939999998 podStartE2EDuration="5.551413629s" totalImagesPullingTime="3.641279435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:30.550956315 +0000 UTC m=+573.103132619" watchObservedRunningTime="2026-10-30 21:59:30.551413629 +0000 UTC m=+573.103589936" Oct 30 21:59:41.225954 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:41.225923 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-bg2f2" Oct 30 21:59:57.419734 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:57.419694 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 21:59:57.660282 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:57.660218 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 21:59:57.660478 ip-10-0-135-7 kubenswrapper[6006]: I1030 21:59:57.660302 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 21:59:57.934384 ip-10-0-135-7 kubenswrapper[6006]: E1030 21:59:57.934330 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:00:00.181855 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:00.181814 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb"] Oct 30 22:00:00.185417 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:00.185389 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb" Oct 30 22:00:00.235600 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:00.235561 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:00:00.235600 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:00.235593 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:00:00.253125 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:00.253084 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hssvj\" (UniqueName: \"kubernetes.io/projected/58e92367-358e-4395-96e6-114d27182004-kube-api-access-hssvj\") pod \"maas-api-key-cleanup-29889960-vhxjb\" (UID: \"58e92367-358e-4395-96e6-114d27182004\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb" Oct 30 22:00:00.253125 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:00.253140 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e92367-358e-4395-96e6-114d27182004-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-vhxjb\" (UID: \"58e92367-358e-4395-96e6-114d27182004\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb" Oct 30 22:00:00.282568 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:00.282534 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:00:00.284088 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:00.284055 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb"] Oct 30 22:00:00.353993 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:00.353954 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hssvj\" (UniqueName: \"kubernetes.io/projected/58e92367-358e-4395-96e6-114d27182004-kube-api-access-hssvj\") pod \"maas-api-key-cleanup-29889960-vhxjb\" (UID: \"58e92367-358e-4395-96e6-114d27182004\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb" Oct 30 22:00:00.354195 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:00.354001 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e92367-358e-4395-96e6-114d27182004-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-vhxjb\" (UID: \"58e92367-358e-4395-96e6-114d27182004\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb" Oct 30 22:00:00.354702 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:00.354681 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e92367-358e-4395-96e6-114d27182004-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-vhxjb\" (UID: \"58e92367-358e-4395-96e6-114d27182004\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb" Oct 30 22:00:00.426050 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:00.425932 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hssvj\" (UniqueName: \"kubernetes.io/projected/58e92367-358e-4395-96e6-114d27182004-kube-api-access-hssvj\") pod \"maas-api-key-cleanup-29889960-vhxjb\" (UID: \"58e92367-358e-4395-96e6-114d27182004\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb" Oct 30 22:00:00.516863 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:00.515575 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb" Oct 30 22:00:00.890807 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:00.890763 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb"] Oct 30 22:00:00.893295 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:00:00.893262 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e92367_358e_4395_96e6_114d27182004.slice/crio-a0d9a34ac60fe226ac36edb811c058e3dd15f3bb98c2ede856fccdb442814d59 WatchSource:0}: Error finding container a0d9a34ac60fe226ac36edb811c058e3dd15f3bb98c2ede856fccdb442814d59: Status 404 returned error can't find the container with id a0d9a34ac60fe226ac36edb811c058e3dd15f3bb98c2ede856fccdb442814d59 Oct 30 22:00:01.225762 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:01.225667 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wb5v2" Oct 30 22:00:01.228917 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:01.228885 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-68ns6" Oct 30 22:00:01.367223 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:01.367186 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb" event={"ID":"58e92367-358e-4395-96e6-114d27182004","Type":"ContainerStarted","Data":"a0d9a34ac60fe226ac36edb811c058e3dd15f3bb98c2ede856fccdb442814d59"} Oct 30 22:00:02.691051 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:02.691011 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2lsx8"] Oct 30 22:00:02.757114 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:02.757070 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2lsx8"] Oct 30 22:00:02.757295 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:02.757183 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-2lsx8" Oct 30 22:00:02.879960 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:02.879923 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrpvl\" (UniqueName: \"kubernetes.io/projected/503dd600-56d3-49d7-a873-9ba12441fc6e-kube-api-access-xrpvl\") pod \"authorino-6f6944bcdd-2lsx8\" (UID: \"503dd600-56d3-49d7-a873-9ba12441fc6e\") " pod="kuadrant-system/authorino-6f6944bcdd-2lsx8" Oct 30 22:00:02.981283 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:02.981160 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrpvl\" (UniqueName: \"kubernetes.io/projected/503dd600-56d3-49d7-a873-9ba12441fc6e-kube-api-access-xrpvl\") pod \"authorino-6f6944bcdd-2lsx8\" (UID: \"503dd600-56d3-49d7-a873-9ba12441fc6e\") " pod="kuadrant-system/authorino-6f6944bcdd-2lsx8" Oct 30 22:00:03.042759 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:03.042727 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrpvl\" (UniqueName: \"kubernetes.io/projected/503dd600-56d3-49d7-a873-9ba12441fc6e-kube-api-access-xrpvl\") pod \"authorino-6f6944bcdd-2lsx8\" (UID: \"503dd600-56d3-49d7-a873-9ba12441fc6e\") " pod="kuadrant-system/authorino-6f6944bcdd-2lsx8" Oct 30 22:00:03.074337 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:03.074286 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2lsx8"] Oct 30 22:00:03.074612 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:03.074596 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-2lsx8" Oct 30 22:00:03.112944 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:03.112904 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6cff5fd677-fmpgw"] Oct 30 22:00:03.140151 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:03.140121 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cff5fd677-fmpgw" Oct 30 22:00:03.178410 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:03.176808 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cff5fd677-fmpgw"] Oct 30 22:00:03.284975 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:03.284938 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52jg8\" (UniqueName: \"kubernetes.io/projected/cf440d0a-135c-49a9-bb93-2783df3fb076-kube-api-access-52jg8\") pod \"authorino-6cff5fd677-fmpgw\" (UID: \"cf440d0a-135c-49a9-bb93-2783df3fb076\") " pod="kuadrant-system/authorino-6cff5fd677-fmpgw" Oct 30 22:00:03.386078 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:03.386033 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52jg8\" (UniqueName: \"kubernetes.io/projected/cf440d0a-135c-49a9-bb93-2783df3fb076-kube-api-access-52jg8\") pod \"authorino-6cff5fd677-fmpgw\" (UID: \"cf440d0a-135c-49a9-bb93-2783df3fb076\") " pod="kuadrant-system/authorino-6cff5fd677-fmpgw" Oct 30 22:00:03.389164 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:03.389137 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cff5fd677-fmpgw"] Oct 30 22:00:03.389395 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:00:03.389375 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-52jg8], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6cff5fd677-fmpgw" podUID="cf440d0a-135c-49a9-bb93-2783df3fb076" Oct 30 22:00:03.422097 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:03.422063 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-768994dc9f-x4ffp"] Oct 30 22:00:03.451025 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:03.450990 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52jg8\" (UniqueName: \"kubernetes.io/projected/cf440d0a-135c-49a9-bb93-2783df3fb076-kube-api-access-52jg8\") pod \"authorino-6cff5fd677-fmpgw\" (UID: \"cf440d0a-135c-49a9-bb93-2783df3fb076\") " pod="kuadrant-system/authorino-6cff5fd677-fmpgw" Oct 30 22:00:03.454380 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:03.454354 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-768994dc9f-x4ffp" Oct 30 22:00:03.481002 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:03.480968 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-768994dc9f-x4ffp"] Oct 30 22:00:03.494046 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:03.494019 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2lsx8"] Oct 30 22:00:03.496151 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:00:03.496115 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod503dd600_56d3_49d7_a873_9ba12441fc6e.slice/crio-3584df0bea79f7cb4854ffaffc3df91c562a764c77fc8fcdf38900780592cfbc WatchSource:0}: Error finding container 3584df0bea79f7cb4854ffaffc3df91c562a764c77fc8fcdf38900780592cfbc: Status 404 returned error can't find the container with id 3584df0bea79f7cb4854ffaffc3df91c562a764c77fc8fcdf38900780592cfbc Oct 30 22:00:03.504275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:03.504174 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 22:00:03.588537 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:03.588493 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c428efb6-c79d-4516-83f4-4970235c9980-tls-cert\") pod \"authorino-768994dc9f-x4ffp\" (UID: \"c428efb6-c79d-4516-83f4-4970235c9980\") " pod="kuadrant-system/authorino-768994dc9f-x4ffp" Oct 30 22:00:03.588714 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:03.588566 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2gzd\" (UniqueName: \"kubernetes.io/projected/c428efb6-c79d-4516-83f4-4970235c9980-kube-api-access-m2gzd\") pod \"authorino-768994dc9f-x4ffp\" (UID: \"c428efb6-c79d-4516-83f4-4970235c9980\") " pod="kuadrant-system/authorino-768994dc9f-x4ffp" Oct 30 22:00:03.689877 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:03.689829 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c428efb6-c79d-4516-83f4-4970235c9980-tls-cert\") pod \"authorino-768994dc9f-x4ffp\" (UID: \"c428efb6-c79d-4516-83f4-4970235c9980\") " pod="kuadrant-system/authorino-768994dc9f-x4ffp" Oct 30 22:00:03.690038 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:03.689887 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2gzd\" (UniqueName: \"kubernetes.io/projected/c428efb6-c79d-4516-83f4-4970235c9980-kube-api-access-m2gzd\") pod \"authorino-768994dc9f-x4ffp\" (UID: \"c428efb6-c79d-4516-83f4-4970235c9980\") " pod="kuadrant-system/authorino-768994dc9f-x4ffp" Oct 30 22:00:03.692662 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:03.692627 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c428efb6-c79d-4516-83f4-4970235c9980-tls-cert\") pod \"authorino-768994dc9f-x4ffp\" (UID: \"c428efb6-c79d-4516-83f4-4970235c9980\") " pod="kuadrant-system/authorino-768994dc9f-x4ffp" Oct 30 22:00:03.748988 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:03.748948 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2gzd\" (UniqueName: \"kubernetes.io/projected/c428efb6-c79d-4516-83f4-4970235c9980-kube-api-access-m2gzd\") pod \"authorino-768994dc9f-x4ffp\" (UID: \"c428efb6-c79d-4516-83f4-4970235c9980\") " pod="kuadrant-system/authorino-768994dc9f-x4ffp" Oct 30 22:00:03.790972 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:03.790923 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-768994dc9f-x4ffp" Oct 30 22:00:04.169617 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:04.169588 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-768994dc9f-x4ffp"] Oct 30 22:00:04.171338 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:00:04.171292 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc428efb6_c79d_4516_83f4_4970235c9980.slice/crio-9b51ba8da3feb00eb671db087e4030932d2941972053e0d10df25a6c16b22990 WatchSource:0}: Error finding container 9b51ba8da3feb00eb671db087e4030932d2941972053e0d10df25a6c16b22990: Status 404 returned error can't find the container with id 9b51ba8da3feb00eb671db087e4030932d2941972053e0d10df25a6c16b22990 Oct 30 22:00:04.384344 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:04.384190 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-768994dc9f-x4ffp" event={"ID":"c428efb6-c79d-4516-83f4-4970235c9980","Type":"ContainerStarted","Data":"9b51ba8da3feb00eb671db087e4030932d2941972053e0d10df25a6c16b22990"} Oct 30 22:00:04.386150 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:04.386123 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-2lsx8" event={"ID":"503dd600-56d3-49d7-a873-9ba12441fc6e","Type":"ContainerStarted","Data":"3584df0bea79f7cb4854ffaffc3df91c562a764c77fc8fcdf38900780592cfbc"} Oct 30 22:00:04.386307 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:04.386158 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cff5fd677-fmpgw" Oct 30 22:00:04.396465 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:04.396438 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cff5fd677-fmpgw" Oct 30 22:00:04.497339 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:04.497233 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf440d0a-135c-49a9-bb93-2783df3fb076-kube-api-access-52jg8\" (UniqueName: \"kubernetes.io/projected/cf440d0a-135c-49a9-bb93-2783df3fb076-kube-api-access-52jg8\") pod \"cf440d0a-135c-49a9-bb93-2783df3fb076\" (UID: \"cf440d0a-135c-49a9-bb93-2783df3fb076\") " Oct 30 22:00:04.499520 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:04.499483 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf440d0a-135c-49a9-bb93-2783df3fb076-kube-api-access-52jg8" pod "cf440d0a-135c-49a9-bb93-2783df3fb076" (UID: "cf440d0a-135c-49a9-bb93-2783df3fb076"). InnerVolumeSpecName "kube-api-access-52jg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:04.599185 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:04.599128 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52jg8\" (UniqueName: \"kubernetes.io/projected/cf440d0a-135c-49a9-bb93-2783df3fb076-kube-api-access-52jg8\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:05.391486 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:05.391455 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cff5fd677-fmpgw" Oct 30 22:00:05.618151 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:05.618106 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cff5fd677-fmpgw"] Oct 30 22:00:05.624378 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:05.624346 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6cff5fd677-fmpgw"] Oct 30 22:00:06.398074 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:06.398038 6006 generic.go:356] "Generic (PLEG): container finished" podID="b77193b8-0b50-4d53-beb9-619be2ea30b5" containerID="1a4656ab03bb4e27ff95a8e1f968c1faaccb8aa82288c28504bc0dfb9d32e167" exitCode=0 Oct 30 22:00:06.398529 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:06.398105 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" event={"ID":"b77193b8-0b50-4d53-beb9-619be2ea30b5","Type":"ContainerDied","Data":"1a4656ab03bb4e27ff95a8e1f968c1faaccb8aa82288c28504bc0dfb9d32e167"} Oct 30 22:00:06.398572 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:06.398544 6006 scope.go:122] "RemoveContainer" containerID="1a4656ab03bb4e27ff95a8e1f968c1faaccb8aa82288c28504bc0dfb9d32e167" Oct 30 22:00:06.399664 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:06.399634 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:00:06.399724 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:06.399710 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:00:06.399758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:06.399743 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:00:07.430584 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:07.430549 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" event={"ID":"b77193b8-0b50-4d53-beb9-619be2ea30b5","Type":"ContainerStarted","Data":"75d1617e04a28e90cec827062429e342d99d54c6f48e119ade4b7b0665b14c7f"} Oct 30 22:00:07.618847 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:07.618702 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gkxzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 22:00:07.618847 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:07.618771 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" podUID="b77193b8-0b50-4d53-beb9-619be2ea30b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 22:00:07.630466 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:07.630199 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf440d0a-135c-49a9-bb93-2783df3fb076" path="/var/lib/kubelet/pods/cf440d0a-135c-49a9-bb93-2783df3fb076/volumes" Oct 30 22:00:07.630755 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:07.630520 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 22:00:08.439077 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:08.439041 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-768994dc9f-x4ffp" event={"ID":"c428efb6-c79d-4516-83f4-4970235c9980","Type":"ContainerStarted","Data":"76c8f51bc5909757118f033a20e5a3adcd2f6069dd2291da9faec6b9336e5d52"} Oct 30 22:00:08.551591 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:08.551535 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-768994dc9f-x4ffp" podStartSLOduration=2.278302863 podStartE2EDuration="5.551520004s" totalImagesPullingTime="3.273217141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:08.550265316 +0000 UTC m=+611.102441610" watchObservedRunningTime="2026-10-30 22:00:08.551520004 +0000 UTC m=+611.103696317" Oct 30 22:00:08.586344 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:08.585938 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5lrcf"] Oct 30 22:00:08.586882 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:08.586599 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-5lrcf" podUID="c59f6fb6-03b1-44f0-82d6-536728dbb508" containerName="authorino" containerID="cri-o://015984b676650b628a76693da06fd13e228f0901ab7991e7dadb27a4b1bcab6a" gracePeriod=30 Oct 30 22:00:08.615714 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:08.615675 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gkxzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 22:00:08.615981 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:08.615950 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" podUID="b77193b8-0b50-4d53-beb9-619be2ea30b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 22:00:08.893655 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:08.893621 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5lrcf" Oct 30 22:00:09.049774 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:09.049302 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c59f6fb6-03b1-44f0-82d6-536728dbb508-kube-api-access-hpqh8\" (UniqueName: \"kubernetes.io/projected/c59f6fb6-03b1-44f0-82d6-536728dbb508-kube-api-access-hpqh8\") pod \"c59f6fb6-03b1-44f0-82d6-536728dbb508\" (UID: \"c59f6fb6-03b1-44f0-82d6-536728dbb508\") " Oct 30 22:00:09.051594 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:09.051518 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c59f6fb6-03b1-44f0-82d6-536728dbb508-kube-api-access-hpqh8" pod "c59f6fb6-03b1-44f0-82d6-536728dbb508" (UID: "c59f6fb6-03b1-44f0-82d6-536728dbb508"). InnerVolumeSpecName "kube-api-access-hpqh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:09.150831 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:09.150780 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpqh8\" (UniqueName: \"kubernetes.io/projected/c59f6fb6-03b1-44f0-82d6-536728dbb508-kube-api-access-hpqh8\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:09.445113 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:09.445018 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb" event={"ID":"58e92367-358e-4395-96e6-114d27182004","Type":"ContainerStarted","Data":"d71468d0e9880bf47489458b71dcb646996f29ed1ef617cc02d76003b8f762cd"} Oct 30 22:00:09.446842 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:09.446813 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-2lsx8" event={"ID":"503dd600-56d3-49d7-a873-9ba12441fc6e","Type":"ContainerStarted","Data":"cc134008ed9d33f7c2e7127af246f20bf200646f899a08447a8d668859039d09"} Oct 30 22:00:09.446976 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:09.446855 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-2lsx8" podUID="503dd600-56d3-49d7-a873-9ba12441fc6e" containerName="authorino" containerID="cri-o://cc134008ed9d33f7c2e7127af246f20bf200646f899a08447a8d668859039d09" gracePeriod=30 Oct 30 22:00:09.448209 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:09.448183 6006 generic.go:356] "Generic (PLEG): container finished" podID="c59f6fb6-03b1-44f0-82d6-536728dbb508" containerID="015984b676650b628a76693da06fd13e228f0901ab7991e7dadb27a4b1bcab6a" exitCode=0 Oct 30 22:00:09.448318 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:09.448288 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5lrcf" Oct 30 22:00:09.448382 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:09.448322 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5lrcf" event={"ID":"c59f6fb6-03b1-44f0-82d6-536728dbb508","Type":"ContainerDied","Data":"015984b676650b628a76693da06fd13e228f0901ab7991e7dadb27a4b1bcab6a"} Oct 30 22:00:09.448441 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:09.448379 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5lrcf" event={"ID":"c59f6fb6-03b1-44f0-82d6-536728dbb508","Type":"ContainerDied","Data":"d84d4bc895448b52d62c44b9817db851c1f7d8e46bcf36381666a77129caaeb0"} Oct 30 22:00:09.448441 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:09.448405 6006 scope.go:122] "RemoveContainer" containerID="015984b676650b628a76693da06fd13e228f0901ab7991e7dadb27a4b1bcab6a" Oct 30 22:00:09.457634 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:09.457600 6006 scope.go:122] "RemoveContainer" containerID="015984b676650b628a76693da06fd13e228f0901ab7991e7dadb27a4b1bcab6a" Oct 30 22:00:09.457896 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:00:09.457878 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"015984b676650b628a76693da06fd13e228f0901ab7991e7dadb27a4b1bcab6a\": container with ID starting with 015984b676650b628a76693da06fd13e228f0901ab7991e7dadb27a4b1bcab6a not found: ID does not exist" containerID="015984b676650b628a76693da06fd13e228f0901ab7991e7dadb27a4b1bcab6a" Oct 30 22:00:09.457964 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:09.457904 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"015984b676650b628a76693da06fd13e228f0901ab7991e7dadb27a4b1bcab6a"} err="failed to get container status \"015984b676650b628a76693da06fd13e228f0901ab7991e7dadb27a4b1bcab6a\": rpc error: code = NotFound desc = could not find container \"015984b676650b628a76693da06fd13e228f0901ab7991e7dadb27a4b1bcab6a\": container with ID starting with 015984b676650b628a76693da06fd13e228f0901ab7991e7dadb27a4b1bcab6a not found: ID does not exist" Oct 30 22:00:09.564018 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:09.563952 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb" podStartSLOduration=2.068603408 podStartE2EDuration="9.56393809s" totalImagesPullingTime="7.495334682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:09.56239157 +0000 UTC m=+612.114567864" watchObservedRunningTime="2026-10-30 22:00:09.56393809 +0000 UTC m=+612.116114389" Oct 30 22:00:09.620501 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:09.615767 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gkxzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 22:00:09.620501 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:09.615827 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" podUID="b77193b8-0b50-4d53-beb9-619be2ea30b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 22:00:09.688934 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:09.688881 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-2lsx8" podStartSLOduration=2.028999797 podStartE2EDuration="7.688866011s" totalImagesPullingTime="5.659866214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:09.686277405 +0000 UTC m=+612.238453706" watchObservedRunningTime="2026-10-30 22:00:09.688866011 +0000 UTC m=+612.241042310" Oct 30 22:00:09.755751 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:09.755722 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-2lsx8" Oct 30 22:00:09.824296 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:09.824082 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5lrcf"] Oct 30 22:00:09.826050 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:09.826016 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5lrcf"] Oct 30 22:00:09.856328 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:09.856296 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/503dd600-56d3-49d7-a873-9ba12441fc6e-kube-api-access-xrpvl\" (UniqueName: \"kubernetes.io/projected/503dd600-56d3-49d7-a873-9ba12441fc6e-kube-api-access-xrpvl\") pod \"503dd600-56d3-49d7-a873-9ba12441fc6e\" (UID: \"503dd600-56d3-49d7-a873-9ba12441fc6e\") " Oct 30 22:00:09.858708 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:09.858670 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/503dd600-56d3-49d7-a873-9ba12441fc6e-kube-api-access-xrpvl" pod "503dd600-56d3-49d7-a873-9ba12441fc6e" (UID: "503dd600-56d3-49d7-a873-9ba12441fc6e"). InnerVolumeSpecName "kube-api-access-xrpvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:09.957868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:09.957833 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrpvl\" (UniqueName: \"kubernetes.io/projected/503dd600-56d3-49d7-a873-9ba12441fc6e-kube-api-access-xrpvl\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:10.375072 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:10.375030 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 22:00:10.456687 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:10.456649 6006 generic.go:356] "Generic (PLEG): container finished" podID="58e92367-358e-4395-96e6-114d27182004" containerID="d71468d0e9880bf47489458b71dcb646996f29ed1ef617cc02d76003b8f762cd" exitCode=60 Oct 30 22:00:10.457156 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:10.456740 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb" event={"ID":"58e92367-358e-4395-96e6-114d27182004","Type":"ContainerDied","Data":"d71468d0e9880bf47489458b71dcb646996f29ed1ef617cc02d76003b8f762cd"} Oct 30 22:00:10.457156 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:10.456973 6006 scope.go:122] "RemoveContainer" containerID="d71468d0e9880bf47489458b71dcb646996f29ed1ef617cc02d76003b8f762cd" Oct 30 22:00:10.458198 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:10.458174 6006 generic.go:356] "Generic (PLEG): container finished" podID="503dd600-56d3-49d7-a873-9ba12441fc6e" containerID="cc134008ed9d33f7c2e7127af246f20bf200646f899a08447a8d668859039d09" exitCode=0 Oct 30 22:00:10.458363 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:10.458227 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-2lsx8" Oct 30 22:00:10.458363 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:10.458233 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-2lsx8" event={"ID":"503dd600-56d3-49d7-a873-9ba12441fc6e","Type":"ContainerDied","Data":"cc134008ed9d33f7c2e7127af246f20bf200646f899a08447a8d668859039d09"} Oct 30 22:00:10.458363 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:10.458276 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-2lsx8" event={"ID":"503dd600-56d3-49d7-a873-9ba12441fc6e","Type":"ContainerDied","Data":"3584df0bea79f7cb4854ffaffc3df91c562a764c77fc8fcdf38900780592cfbc"} Oct 30 22:00:10.458363 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:10.458293 6006 scope.go:122] "RemoveContainer" containerID="cc134008ed9d33f7c2e7127af246f20bf200646f899a08447a8d668859039d09" Oct 30 22:00:10.469805 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:10.469787 6006 scope.go:122] "RemoveContainer" containerID="cc134008ed9d33f7c2e7127af246f20bf200646f899a08447a8d668859039d09" Oct 30 22:00:10.470103 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:00:10.470084 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc134008ed9d33f7c2e7127af246f20bf200646f899a08447a8d668859039d09\": container with ID starting with cc134008ed9d33f7c2e7127af246f20bf200646f899a08447a8d668859039d09 not found: ID does not exist" containerID="cc134008ed9d33f7c2e7127af246f20bf200646f899a08447a8d668859039d09" Oct 30 22:00:10.470184 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:10.470112 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc134008ed9d33f7c2e7127af246f20bf200646f899a08447a8d668859039d09"} err="failed to get container status \"cc134008ed9d33f7c2e7127af246f20bf200646f899a08447a8d668859039d09\": rpc error: code = NotFound desc = could not find container \"cc134008ed9d33f7c2e7127af246f20bf200646f899a08447a8d668859039d09\": container with ID starting with cc134008ed9d33f7c2e7127af246f20bf200646f899a08447a8d668859039d09 not found: ID does not exist" Oct 30 22:00:10.618275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:10.617676 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gkxzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 22:00:10.618275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:10.617746 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" podUID="b77193b8-0b50-4d53-beb9-619be2ea30b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 22:00:10.693314 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:10.693214 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2lsx8"] Oct 30 22:00:10.720274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:10.709871 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2lsx8"] Oct 30 22:00:11.466452 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:11.466402 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb" event={"ID":"58e92367-358e-4395-96e6-114d27182004","Type":"ContainerStarted","Data":"61075ecb851bd52a934b0128cd2537a53ac387c074e79ffa8474c741882d46bf"} Oct 30 22:00:11.617139 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:11.617087 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gkxzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 22:00:11.617506 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:11.617432 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" podUID="b77193b8-0b50-4d53-beb9-619be2ea30b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 22:00:11.627751 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:11.627708 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="503dd600-56d3-49d7-a873-9ba12441fc6e" path="/var/lib/kubelet/pods/503dd600-56d3-49d7-a873-9ba12441fc6e/volumes" Oct 30 22:00:11.628085 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:11.628067 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c59f6fb6-03b1-44f0-82d6-536728dbb508" path="/var/lib/kubelet/pods/c59f6fb6-03b1-44f0-82d6-536728dbb508/volumes" Oct 30 22:00:12.474964 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:12.474931 6006 generic.go:356] "Generic (PLEG): container finished" podID="58e92367-358e-4395-96e6-114d27182004" containerID="61075ecb851bd52a934b0128cd2537a53ac387c074e79ffa8474c741882d46bf" exitCode=60 Oct 30 22:00:12.475415 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:12.475020 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb" event={"ID":"58e92367-358e-4395-96e6-114d27182004","Type":"ContainerDied","Data":"61075ecb851bd52a934b0128cd2537a53ac387c074e79ffa8474c741882d46bf"} Oct 30 22:00:12.475415 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:12.475083 6006 scope.go:122] "RemoveContainer" containerID="d71468d0e9880bf47489458b71dcb646996f29ed1ef617cc02d76003b8f762cd" Oct 30 22:00:12.475415 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:12.475360 6006 scope.go:122] "RemoveContainer" containerID="61075ecb851bd52a934b0128cd2537a53ac387c074e79ffa8474c741882d46bf" Oct 30 22:00:12.475641 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:00:12.475621 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-vhxjb_odh-ai-gateway-infra(58e92367-358e-4395-96e6-114d27182004)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb" podUID="58e92367-358e-4395-96e6-114d27182004" Oct 30 22:00:12.621078 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:12.621047 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 22:00:12.622005 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:12.621983 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 22:00:13.482767 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:13.482735 6006 scope.go:122] "RemoveContainer" containerID="61075ecb851bd52a934b0128cd2537a53ac387c074e79ffa8474c741882d46bf" Oct 30 22:00:13.483160 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:00:13.482935 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-vhxjb_odh-ai-gateway-infra(58e92367-358e-4395-96e6-114d27182004)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb" podUID="58e92367-358e-4395-96e6-114d27182004" Oct 30 22:00:25.596808 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:25.596775 6006 scope.go:122] "RemoveContainer" containerID="61075ecb851bd52a934b0128cd2537a53ac387c074e79ffa8474c741882d46bf" Oct 30 22:00:26.548594 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:26.548559 6006 generic.go:356] "Generic (PLEG): container finished" podID="58e92367-358e-4395-96e6-114d27182004" containerID="61280f19492b7419986a9282e6278081776e9f50f2996e75b2d961e461013ce8" exitCode=60 Oct 30 22:00:26.548759 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:26.548622 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb" event={"ID":"58e92367-358e-4395-96e6-114d27182004","Type":"ContainerDied","Data":"61280f19492b7419986a9282e6278081776e9f50f2996e75b2d961e461013ce8"} Oct 30 22:00:26.548759 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:26.548705 6006 scope.go:122] "RemoveContainer" containerID="61075ecb851bd52a934b0128cd2537a53ac387c074e79ffa8474c741882d46bf" Oct 30 22:00:26.549103 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:26.549084 6006 scope.go:122] "RemoveContainer" containerID="61280f19492b7419986a9282e6278081776e9f50f2996e75b2d961e461013ce8" Oct 30 22:00:26.549379 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:00:26.549360 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-vhxjb_odh-ai-gateway-infra(58e92367-358e-4395-96e6-114d27182004)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb" podUID="58e92367-358e-4395-96e6-114d27182004" Oct 30 22:00:27.721407 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:27.721370 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb"] Oct 30 22:00:27.858858 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:27.858827 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb" Oct 30 22:00:27.914323 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:00:27.914286 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:00:28.031544 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:28.031454 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e92367-358e-4395-96e6-114d27182004-kube-api-access-hssvj\" (UniqueName: \"kubernetes.io/projected/58e92367-358e-4395-96e6-114d27182004-kube-api-access-hssvj\") pod \"58e92367-358e-4395-96e6-114d27182004\" (UID: \"58e92367-358e-4395-96e6-114d27182004\") " Oct 30 22:00:28.031544 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:28.031532 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58e92367-358e-4395-96e6-114d27182004-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e92367-358e-4395-96e6-114d27182004-maas-api-service-ca\") pod \"58e92367-358e-4395-96e6-114d27182004\" (UID: \"58e92367-358e-4395-96e6-114d27182004\") " Oct 30 22:00:28.031968 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:28.031923 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58e92367-358e-4395-96e6-114d27182004-maas-api-service-ca" pod "58e92367-358e-4395-96e6-114d27182004" (UID: "58e92367-358e-4395-96e6-114d27182004"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:28.032111 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:28.032088 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e92367-358e-4395-96e6-114d27182004-maas-api-service-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:28.033900 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:28.033873 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e92367-358e-4395-96e6-114d27182004-kube-api-access-hssvj" pod "58e92367-358e-4395-96e6-114d27182004" (UID: "58e92367-358e-4395-96e6-114d27182004"). InnerVolumeSpecName "kube-api-access-hssvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:28.132619 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:28.132572 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hssvj\" (UniqueName: \"kubernetes.io/projected/58e92367-358e-4395-96e6-114d27182004-kube-api-access-hssvj\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:28.562010 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:28.561973 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb" Oct 30 22:00:28.562010 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:28.562001 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb" event={"ID":"58e92367-358e-4395-96e6-114d27182004","Type":"ContainerDied","Data":"a0d9a34ac60fe226ac36edb811c058e3dd15f3bb98c2ede856fccdb442814d59"} Oct 30 22:00:28.562296 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:28.562072 6006 scope.go:122] "RemoveContainer" containerID="61280f19492b7419986a9282e6278081776e9f50f2996e75b2d961e461013ce8" Oct 30 22:00:28.757833 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:28.757800 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb"] Oct 30 22:00:28.773687 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:28.773658 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-vhxjb"] Oct 30 22:00:29.602181 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:29.602142 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58e92367-358e-4395-96e6-114d27182004" path="/var/lib/kubelet/pods/58e92367-358e-4395-96e6-114d27182004/volumes" Oct 30 22:00:49.997195 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:49.997109 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-4n2tx"] Oct 30 22:00:49.997719 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:49.997569 6006 cpu_manager.go:414] "containerMap: removing container" podUID="c59f6fb6-03b1-44f0-82d6-536728dbb508" containerName="authorino" Oct 30 22:00:49.997719 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:49.997582 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c59f6fb6-03b1-44f0-82d6-536728dbb508" containerName="authorino" Oct 30 22:00:49.997719 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:49.997608 6006 cpu_manager.go:414] "containerMap: removing container" podUID="58e92367-358e-4395-96e6-114d27182004" containerName="cleanup" Oct 30 22:00:49.997719 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:49.997614 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58e92367-358e-4395-96e6-114d27182004" containerName="cleanup" Oct 30 22:00:49.997719 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:49.997622 6006 cpu_manager.go:414] "containerMap: removing container" podUID="58e92367-358e-4395-96e6-114d27182004" containerName="cleanup" Oct 30 22:00:49.997719 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:49.997627 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58e92367-358e-4395-96e6-114d27182004" containerName="cleanup" Oct 30 22:00:49.997719 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:49.997644 6006 cpu_manager.go:414] "containerMap: removing container" podUID="503dd600-56d3-49d7-a873-9ba12441fc6e" containerName="authorino" Oct 30 22:00:49.997719 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:49.997649 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="503dd600-56d3-49d7-a873-9ba12441fc6e" containerName="authorino" Oct 30 22:00:49.998010 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:49.997736 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="c59f6fb6-03b1-44f0-82d6-536728dbb508" containerName="authorino" Oct 30 22:00:49.998010 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:49.997746 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="58e92367-358e-4395-96e6-114d27182004" containerName="cleanup" Oct 30 22:00:49.998010 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:49.997751 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="58e92367-358e-4395-96e6-114d27182004" containerName="cleanup" Oct 30 22:00:49.998010 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:49.997759 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="503dd600-56d3-49d7-a873-9ba12441fc6e" containerName="authorino" Oct 30 22:00:50.005661 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:50.005636 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-4n2tx" Oct 30 22:00:50.057077 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:50.057039 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-4n2tx"] Oct 30 22:00:50.133044 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:50.133002 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e451335-7336-48fb-b23c-f8b30de7fd33-cert\") pod \"llmisvc-controller-manager-6466c554d8-4n2tx\" (UID: \"5e451335-7336-48fb-b23c-f8b30de7fd33\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-4n2tx" Oct 30 22:00:50.133222 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:50.133068 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f48xn\" (UniqueName: \"kubernetes.io/projected/5e451335-7336-48fb-b23c-f8b30de7fd33-kube-api-access-f48xn\") pod \"llmisvc-controller-manager-6466c554d8-4n2tx\" (UID: \"5e451335-7336-48fb-b23c-f8b30de7fd33\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-4n2tx" Oct 30 22:00:50.234377 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:50.234329 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e451335-7336-48fb-b23c-f8b30de7fd33-cert\") pod \"llmisvc-controller-manager-6466c554d8-4n2tx\" (UID: \"5e451335-7336-48fb-b23c-f8b30de7fd33\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-4n2tx" Oct 30 22:00:50.234554 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:50.234410 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f48xn\" (UniqueName: \"kubernetes.io/projected/5e451335-7336-48fb-b23c-f8b30de7fd33-kube-api-access-f48xn\") pod \"llmisvc-controller-manager-6466c554d8-4n2tx\" (UID: \"5e451335-7336-48fb-b23c-f8b30de7fd33\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-4n2tx" Oct 30 22:00:50.236844 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:50.236816 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e451335-7336-48fb-b23c-f8b30de7fd33-cert\") pod \"llmisvc-controller-manager-6466c554d8-4n2tx\" (UID: \"5e451335-7336-48fb-b23c-f8b30de7fd33\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-4n2tx" Oct 30 22:00:50.293632 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:50.293598 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f48xn\" (UniqueName: \"kubernetes.io/projected/5e451335-7336-48fb-b23c-f8b30de7fd33-kube-api-access-f48xn\") pod \"llmisvc-controller-manager-6466c554d8-4n2tx\" (UID: \"5e451335-7336-48fb-b23c-f8b30de7fd33\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-4n2tx" Oct 30 22:00:50.320270 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:50.320188 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-4n2tx" Oct 30 22:00:50.687228 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:50.687201 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-4n2tx"] Oct 30 22:00:50.689984 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:00:50.689950 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e451335_7336_48fb_b23c_f8b30de7fd33.slice/crio-d0594e8f7ff0e60aa1b8b2098a1ae1d71b050231bc61ca44d0072551e8a349a5 WatchSource:0}: Error finding container d0594e8f7ff0e60aa1b8b2098a1ae1d71b050231bc61ca44d0072551e8a349a5: Status 404 returned error can't find the container with id d0594e8f7ff0e60aa1b8b2098a1ae1d71b050231bc61ca44d0072551e8a349a5 Oct 30 22:00:51.677661 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:51.677623 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-4n2tx" event={"ID":"5e451335-7336-48fb-b23c-f8b30de7fd33","Type":"ContainerStarted","Data":"547a610cc512eea63400bc24244254b4de4c1fe6c0321e2f7ac74c512db05103"} Oct 30 22:00:51.678222 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:51.678188 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-4n2tx" event={"ID":"5e451335-7336-48fb-b23c-f8b30de7fd33","Type":"ContainerStarted","Data":"d0594e8f7ff0e60aa1b8b2098a1ae1d71b050231bc61ca44d0072551e8a349a5"} Oct 30 22:00:51.678436 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:51.678412 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-4n2tx" Oct 30 22:00:51.790438 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:00:51.790381 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-4n2tx" podStartSLOduration=1.912897872 podStartE2EDuration="2.790366169s" totalImagesPullingTime="877.468297ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:51.789553423 +0000 UTC m=+654.341729727" watchObservedRunningTime="2026-10-30 22:00:51.790366169 +0000 UTC m=+654.342542469" Oct 30 22:00:57.911443 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:00:57.911403 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:01:23.690400 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:01:23.690366 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-4n2tx" Oct 30 22:01:23.851989 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:01:23.851949 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-wb5v2"] Oct 30 22:01:23.852226 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:01:23.852197 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wb5v2" podUID="48d0218e-59fe-4f51-9b47-df84046b0486" containerName="manager" containerID="cri-o://38d0d90ec30518b5d084071a90c909f7291af7011310ca0c104a7ea3ed5c4c5c" gracePeriod=30 Oct 30 22:01:24.111628 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:01:24.111595 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wb5v2" Oct 30 22:01:24.139699 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:01:24.139656 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48d0218e-59fe-4f51-9b47-df84046b0486-cert\" (UniqueName: \"kubernetes.io/secret/48d0218e-59fe-4f51-9b47-df84046b0486-cert\") pod \"48d0218e-59fe-4f51-9b47-df84046b0486\" (UID: \"48d0218e-59fe-4f51-9b47-df84046b0486\") " Oct 30 22:01:24.139890 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:01:24.139711 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48d0218e-59fe-4f51-9b47-df84046b0486-kube-api-access-wk7d2\" (UniqueName: \"kubernetes.io/projected/48d0218e-59fe-4f51-9b47-df84046b0486-kube-api-access-wk7d2\") pod \"48d0218e-59fe-4f51-9b47-df84046b0486\" (UID: \"48d0218e-59fe-4f51-9b47-df84046b0486\") " Oct 30 22:01:24.141933 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:01:24.141904 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48d0218e-59fe-4f51-9b47-df84046b0486-cert" pod "48d0218e-59fe-4f51-9b47-df84046b0486" (UID: "48d0218e-59fe-4f51-9b47-df84046b0486"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:24.142061 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:01:24.142043 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d0218e-59fe-4f51-9b47-df84046b0486-kube-api-access-wk7d2" pod "48d0218e-59fe-4f51-9b47-df84046b0486" (UID: "48d0218e-59fe-4f51-9b47-df84046b0486"). InnerVolumeSpecName "kube-api-access-wk7d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:24.240642 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:01:24.240558 6006 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48d0218e-59fe-4f51-9b47-df84046b0486-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:24.240642 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:01:24.240587 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk7d2\" (UniqueName: \"kubernetes.io/projected/48d0218e-59fe-4f51-9b47-df84046b0486-kube-api-access-wk7d2\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:24.845382 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:01:24.845345 6006 generic.go:356] "Generic (PLEG): container finished" podID="48d0218e-59fe-4f51-9b47-df84046b0486" containerID="38d0d90ec30518b5d084071a90c909f7291af7011310ca0c104a7ea3ed5c4c5c" exitCode=0 Oct 30 22:01:24.845785 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:01:24.845392 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wb5v2" Oct 30 22:01:24.845785 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:01:24.845430 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wb5v2" event={"ID":"48d0218e-59fe-4f51-9b47-df84046b0486","Type":"ContainerDied","Data":"38d0d90ec30518b5d084071a90c909f7291af7011310ca0c104a7ea3ed5c4c5c"} Oct 30 22:01:24.845785 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:01:24.845485 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wb5v2" event={"ID":"48d0218e-59fe-4f51-9b47-df84046b0486","Type":"ContainerDied","Data":"e9719d856ea0d71ed6d1e83e391f2ea6ef3fafbe3e6000b643d06518b500a38b"} Oct 30 22:01:24.845785 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:01:24.845508 6006 scope.go:122] "RemoveContainer" containerID="38d0d90ec30518b5d084071a90c909f7291af7011310ca0c104a7ea3ed5c4c5c" Oct 30 22:01:24.855135 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:01:24.854946 6006 scope.go:122] "RemoveContainer" containerID="38d0d90ec30518b5d084071a90c909f7291af7011310ca0c104a7ea3ed5c4c5c" Oct 30 22:01:24.855288 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:01:24.855266 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38d0d90ec30518b5d084071a90c909f7291af7011310ca0c104a7ea3ed5c4c5c\": container with ID starting with 38d0d90ec30518b5d084071a90c909f7291af7011310ca0c104a7ea3ed5c4c5c not found: ID does not exist" containerID="38d0d90ec30518b5d084071a90c909f7291af7011310ca0c104a7ea3ed5c4c5c" Oct 30 22:01:24.855369 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:01:24.855301 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d0d90ec30518b5d084071a90c909f7291af7011310ca0c104a7ea3ed5c4c5c"} err="failed to get container status \"38d0d90ec30518b5d084071a90c909f7291af7011310ca0c104a7ea3ed5c4c5c\": rpc error: code = NotFound desc = could not find container \"38d0d90ec30518b5d084071a90c909f7291af7011310ca0c104a7ea3ed5c4c5c\": container with ID starting with 38d0d90ec30518b5d084071a90c909f7291af7011310ca0c104a7ea3ed5c4c5c not found: ID does not exist" Oct 30 22:01:25.003522 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:01:25.003488 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-wb5v2"] Oct 30 22:01:25.050445 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:01:25.050397 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-wb5v2"] Oct 30 22:01:25.602396 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:01:25.602361 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48d0218e-59fe-4f51-9b47-df84046b0486" path="/var/lib/kubelet/pods/48d0218e-59fe-4f51-9b47-df84046b0486/volumes" Oct 30 22:01:27.910180 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:01:27.910144 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:01:48.257523 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:01:48.257482 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:01:57.916392 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:01:57.916344 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:01:58.836404 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:01:58.833118 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:02:13.453477 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:13.453390 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:02:26.391837 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:26.391799 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:02:27.910500 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:02:27.910462 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:02:41.182130 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:41.182090 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-667cc846fb-7smlq"] Oct 30 22:02:41.182743 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:41.182720 6006 cpu_manager.go:414] "containerMap: removing container" podUID="58e92367-358e-4395-96e6-114d27182004" containerName="cleanup" Oct 30 22:02:41.182827 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:41.182745 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58e92367-358e-4395-96e6-114d27182004" containerName="cleanup" Oct 30 22:02:41.182827 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:41.182774 6006 cpu_manager.go:414] "containerMap: removing container" podUID="48d0218e-59fe-4f51-9b47-df84046b0486" containerName="manager" Oct 30 22:02:41.182827 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:41.182783 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48d0218e-59fe-4f51-9b47-df84046b0486" containerName="manager" Oct 30 22:02:41.182977 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:41.182886 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="58e92367-358e-4395-96e6-114d27182004" containerName="cleanup" Oct 30 22:02:41.182977 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:41.182903 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="48d0218e-59fe-4f51-9b47-df84046b0486" containerName="manager" Oct 30 22:02:41.186026 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:41.186004 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-667cc846fb-7smlq" Oct 30 22:02:41.234343 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:41.234305 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-667cc846fb-7smlq"] Oct 30 22:02:41.282460 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:41.282410 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c328f12f-dbd3-4d02-b759-08a11ab99993-tls-cert\") pod \"authorino-667cc846fb-7smlq\" (UID: \"c328f12f-dbd3-4d02-b759-08a11ab99993\") " pod="kuadrant-system/authorino-667cc846fb-7smlq" Oct 30 22:02:41.282648 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:41.282548 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct76b\" (UniqueName: \"kubernetes.io/projected/c328f12f-dbd3-4d02-b759-08a11ab99993-kube-api-access-ct76b\") pod \"authorino-667cc846fb-7smlq\" (UID: \"c328f12f-dbd3-4d02-b759-08a11ab99993\") " pod="kuadrant-system/authorino-667cc846fb-7smlq" Oct 30 22:02:41.383085 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:41.383041 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c328f12f-dbd3-4d02-b759-08a11ab99993-tls-cert\") pod \"authorino-667cc846fb-7smlq\" (UID: \"c328f12f-dbd3-4d02-b759-08a11ab99993\") " pod="kuadrant-system/authorino-667cc846fb-7smlq" Oct 30 22:02:41.383303 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:41.383106 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct76b\" (UniqueName: \"kubernetes.io/projected/c328f12f-dbd3-4d02-b759-08a11ab99993-kube-api-access-ct76b\") pod \"authorino-667cc846fb-7smlq\" (UID: \"c328f12f-dbd3-4d02-b759-08a11ab99993\") " pod="kuadrant-system/authorino-667cc846fb-7smlq" Oct 30 22:02:41.385571 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:41.385544 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c328f12f-dbd3-4d02-b759-08a11ab99993-tls-cert\") pod \"authorino-667cc846fb-7smlq\" (UID: \"c328f12f-dbd3-4d02-b759-08a11ab99993\") " pod="kuadrant-system/authorino-667cc846fb-7smlq" Oct 30 22:02:41.442243 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:41.442159 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct76b\" (UniqueName: \"kubernetes.io/projected/c328f12f-dbd3-4d02-b759-08a11ab99993-kube-api-access-ct76b\") pod \"authorino-667cc846fb-7smlq\" (UID: \"c328f12f-dbd3-4d02-b759-08a11ab99993\") " pod="kuadrant-system/authorino-667cc846fb-7smlq" Oct 30 22:02:41.497997 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:41.497965 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-667cc846fb-7smlq" Oct 30 22:02:41.866622 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:41.866467 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-667cc846fb-7smlq"] Oct 30 22:02:41.869120 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:02:41.869075 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc328f12f_dbd3_4d02_b759_08a11ab99993.slice/crio-2b2d10dd7e5d13a980d3034c8caffb5e34d0ff09336ede41dc830960ef859833 WatchSource:0}: Error finding container 2b2d10dd7e5d13a980d3034c8caffb5e34d0ff09336ede41dc830960ef859833: Status 404 returned error can't find the container with id 2b2d10dd7e5d13a980d3034c8caffb5e34d0ff09336ede41dc830960ef859833 Oct 30 22:02:41.870747 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:41.870726 6006 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:02:42.245933 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:42.245839 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-667cc846fb-7smlq" event={"ID":"c328f12f-dbd3-4d02-b759-08a11ab99993","Type":"ContainerStarted","Data":"2b2d10dd7e5d13a980d3034c8caffb5e34d0ff09336ede41dc830960ef859833"} Oct 30 22:02:44.257089 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:44.257049 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-667cc846fb-7smlq" event={"ID":"c328f12f-dbd3-4d02-b759-08a11ab99993","Type":"ContainerStarted","Data":"75cb1c3b53bd4e0c21cc1fce1b29d9fa01f3eae47164c02b9fac6bce39ea52bc"} Oct 30 22:02:44.380163 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:44.380108 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-667cc846fb-7smlq" podStartSLOduration=1.741662313 podStartE2EDuration="3.380087625s" totalImagesPullingTime="1.638425312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:44.378227438 +0000 UTC m=+766.930403726" watchObservedRunningTime="2026-10-30 22:02:44.380087625 +0000 UTC m=+766.932263929" Oct 30 22:02:44.427403 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:44.427363 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-768994dc9f-x4ffp"] Oct 30 22:02:44.427607 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:44.427581 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-768994dc9f-x4ffp" podUID="c428efb6-c79d-4516-83f4-4970235c9980" containerName="authorino" containerID="cri-o://76c8f51bc5909757118f033a20e5a3adcd2f6069dd2291da9faec6b9336e5d52" gracePeriod=30 Oct 30 22:02:44.708042 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:44.708009 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-768994dc9f-x4ffp" Oct 30 22:02:44.712041 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:44.712012 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c428efb6-c79d-4516-83f4-4970235c9980-kube-api-access-m2gzd\" (UniqueName: \"kubernetes.io/projected/c428efb6-c79d-4516-83f4-4970235c9980-kube-api-access-m2gzd\") pod \"c428efb6-c79d-4516-83f4-4970235c9980\" (UID: \"c428efb6-c79d-4516-83f4-4970235c9980\") " Oct 30 22:02:44.712190 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:44.712153 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c428efb6-c79d-4516-83f4-4970235c9980-tls-cert\" (UniqueName: \"kubernetes.io/secret/c428efb6-c79d-4516-83f4-4970235c9980-tls-cert\") pod \"c428efb6-c79d-4516-83f4-4970235c9980\" (UID: \"c428efb6-c79d-4516-83f4-4970235c9980\") " Oct 30 22:02:44.714277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:44.714218 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c428efb6-c79d-4516-83f4-4970235c9980-kube-api-access-m2gzd" pod "c428efb6-c79d-4516-83f4-4970235c9980" (UID: "c428efb6-c79d-4516-83f4-4970235c9980"). InnerVolumeSpecName "kube-api-access-m2gzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:44.723525 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:44.723494 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c428efb6-c79d-4516-83f4-4970235c9980-tls-cert" pod "c428efb6-c79d-4516-83f4-4970235c9980" (UID: "c428efb6-c79d-4516-83f4-4970235c9980"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:44.815328 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:44.814792 6006 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c428efb6-c79d-4516-83f4-4970235c9980-tls-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:44.815328 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:44.814837 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2gzd\" (UniqueName: \"kubernetes.io/projected/c428efb6-c79d-4516-83f4-4970235c9980-kube-api-access-m2gzd\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.264882 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:45.264779 6006 generic.go:356] "Generic (PLEG): container finished" podID="c428efb6-c79d-4516-83f4-4970235c9980" containerID="76c8f51bc5909757118f033a20e5a3adcd2f6069dd2291da9faec6b9336e5d52" exitCode=0 Oct 30 22:02:45.264882 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:45.264826 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-768994dc9f-x4ffp" Oct 30 22:02:45.264882 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:45.264862 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-768994dc9f-x4ffp" event={"ID":"c428efb6-c79d-4516-83f4-4970235c9980","Type":"ContainerDied","Data":"76c8f51bc5909757118f033a20e5a3adcd2f6069dd2291da9faec6b9336e5d52"} Oct 30 22:02:45.265517 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:45.264903 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-768994dc9f-x4ffp" event={"ID":"c428efb6-c79d-4516-83f4-4970235c9980","Type":"ContainerDied","Data":"9b51ba8da3feb00eb671db087e4030932d2941972053e0d10df25a6c16b22990"} Oct 30 22:02:45.265517 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:45.264929 6006 scope.go:122] "RemoveContainer" containerID="76c8f51bc5909757118f033a20e5a3adcd2f6069dd2291da9faec6b9336e5d52" Oct 30 22:02:45.274827 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:45.274804 6006 scope.go:122] "RemoveContainer" containerID="76c8f51bc5909757118f033a20e5a3adcd2f6069dd2291da9faec6b9336e5d52" Oct 30 22:02:45.275159 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:02:45.275138 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76c8f51bc5909757118f033a20e5a3adcd2f6069dd2291da9faec6b9336e5d52\": container with ID starting with 76c8f51bc5909757118f033a20e5a3adcd2f6069dd2291da9faec6b9336e5d52 not found: ID does not exist" containerID="76c8f51bc5909757118f033a20e5a3adcd2f6069dd2291da9faec6b9336e5d52" Oct 30 22:02:45.275272 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:45.275167 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76c8f51bc5909757118f033a20e5a3adcd2f6069dd2291da9faec6b9336e5d52"} err="failed to get container status \"76c8f51bc5909757118f033a20e5a3adcd2f6069dd2291da9faec6b9336e5d52\": rpc error: code = NotFound desc = could not find container \"76c8f51bc5909757118f033a20e5a3adcd2f6069dd2291da9faec6b9336e5d52\": container with ID starting with 76c8f51bc5909757118f033a20e5a3adcd2f6069dd2291da9faec6b9336e5d52 not found: ID does not exist" Oct 30 22:02:45.420637 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:45.420604 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-768994dc9f-x4ffp"] Oct 30 22:02:45.426178 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:45.426148 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-768994dc9f-x4ffp"] Oct 30 22:02:45.602894 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:02:45.602862 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c428efb6-c79d-4516-83f4-4970235c9980" path="/var/lib/kubelet/pods/c428efb6-c79d-4516-83f4-4970235c9980/volumes" Oct 30 22:02:57.910068 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:02:57.910030 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:03:21.798684 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.798651 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8"] Oct 30 22:03:21.799197 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.799034 6006 cpu_manager.go:414] "containerMap: removing container" podUID="c428efb6-c79d-4516-83f4-4970235c9980" containerName="authorino" Oct 30 22:03:21.799197 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.799044 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c428efb6-c79d-4516-83f4-4970235c9980" containerName="authorino" Oct 30 22:03:21.799197 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.799127 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="c428efb6-c79d-4516-83f4-4970235c9980" containerName="authorino" Oct 30 22:03:21.802228 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.802212 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:21.877429 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.877383 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:21.877637 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.877441 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnm5x\" (UniqueName: \"kubernetes.io/projected/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-kube-api-access-bnm5x\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:21.877637 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.877555 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:21.877762 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.877646 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:21.877762 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.877718 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:21.877846 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.877780 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:21.877846 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.877826 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:21.877923 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.877902 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:21.877957 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.877935 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:21.978620 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.978586 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:21.978783 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.978637 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:21.978783 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.978748 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:21.978858 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.978794 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:21.978858 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.978823 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:21.978858 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.978849 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:21.978995 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.978887 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:21.978995 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.978918 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnm5x\" (UniqueName: \"kubernetes.io/projected/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-kube-api-access-bnm5x\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:21.978995 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.978949 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:21.978995 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.978986 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:21.979201 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.979041 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:21.979201 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.979172 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:21.979399 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.979378 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:21.979604 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.979585 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:21.981083 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.981062 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:21.981362 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:21.981343 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:22.001145 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:22.001115 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8"] Oct 30 22:03:22.039052 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:22.039007 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:22.039215 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:22.039130 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnm5x\" (UniqueName: \"kubernetes.io/projected/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-kube-api-access-bnm5x\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sn4t8\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:22.114841 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:22.114747 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:22.533947 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:22.533867 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8"] Oct 30 22:03:22.566541 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:03:22.566513 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ae69d7_186a_4ff2_bde0_e95e0bc5bb51.slice/crio-4048914dda616008a4ff94dea99ad8da7b9dff94df226f31a1892f9ec5e2150a WatchSource:0}: Error finding container 4048914dda616008a4ff94dea99ad8da7b9dff94df226f31a1892f9ec5e2150a: Status 404 returned error can't find the container with id 4048914dda616008a4ff94dea99ad8da7b9dff94df226f31a1892f9ec5e2150a Oct 30 22:03:22.568629 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:22.568592 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:03:22.568722 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:22.568660 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:03:22.568722 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:22.568692 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:03:23.468181 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:23.468142 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" event={"ID":"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51","Type":"ContainerStarted","Data":"a43ea638943f8fbae09b1efced9f4ac0463316723cab5ad1f16e294e51833db7"} Oct 30 22:03:23.468623 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:23.468200 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" event={"ID":"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51","Type":"ContainerStarted","Data":"4048914dda616008a4ff94dea99ad8da7b9dff94df226f31a1892f9ec5e2150a"} Oct 30 22:03:23.632706 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:23.632654 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" podStartSLOduration=2.632638114 podStartE2EDuration="2.632638114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:23.631231963 +0000 UTC m=+806.183408273" watchObservedRunningTime="2026-10-30 22:03:23.632638114 +0000 UTC m=+806.184814413" Oct 30 22:03:24.122274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:24.118642 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:24.122274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:24.121029 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-sn4t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:03:24.122274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:24.121084 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" podUID="b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 22:03:25.123113 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:25.123072 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-sn4t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:03:25.123624 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:25.123368 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" podUID="b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 22:03:26.116705 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:26.116662 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-sn4t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:03:26.117010 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:26.116979 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" podUID="b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 22:03:27.116719 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:27.116676 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-sn4t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:03:27.117398 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:27.117361 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" podUID="b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 22:03:27.934129 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:03:27.934087 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:03:28.116693 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:28.116653 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-sn4t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:03:28.117002 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:28.116970 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" podUID="b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 22:03:29.115293 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:29.115228 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-sn4t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:03:29.115531 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:29.115311 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" podUID="b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 22:03:30.891805 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:30.891767 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:30.892332 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:30.892192 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:30.893221 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:03:30.893199 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:03:57.910092 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:03:57.910010 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:04:01.691919 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:01.691881 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99"] Oct 30 22:04:01.696361 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:01.696336 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99" Oct 30 22:04:01.747119 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:01.747078 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-a00d9b\"" Oct 30 22:04:01.793670 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:01.793621 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfc98\" (UniqueName: \"kubernetes.io/projected/f9a2a7a5-468b-43f3-8528-1cc78b307324-kube-api-access-lfc98\") pod \"payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99\" (UID: \"f9a2a7a5-468b-43f3-8528-1cc78b307324\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99" Oct 30 22:04:01.793860 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:01.793741 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a2a7a5-468b-43f3-8528-1cc78b307324-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99\" (UID: \"f9a2a7a5-468b-43f3-8528-1cc78b307324\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99" Oct 30 22:04:01.793907 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:01.793880 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-a00d9b-dockercfg-9d5qh\"" Oct 30 22:04:01.795625 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:01.795598 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99"] Oct 30 22:04:01.894804 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:01.894766 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfc98\" (UniqueName: \"kubernetes.io/projected/f9a2a7a5-468b-43f3-8528-1cc78b307324-kube-api-access-lfc98\") pod \"payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99\" (UID: \"f9a2a7a5-468b-43f3-8528-1cc78b307324\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99" Oct 30 22:04:01.894993 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:01.894812 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a2a7a5-468b-43f3-8528-1cc78b307324-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99\" (UID: \"f9a2a7a5-468b-43f3-8528-1cc78b307324\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99" Oct 30 22:04:01.895645 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:01.895617 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a2a7a5-468b-43f3-8528-1cc78b307324-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99\" (UID: \"f9a2a7a5-468b-43f3-8528-1cc78b307324\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99" Oct 30 22:04:01.964839 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:01.964747 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfc98\" (UniqueName: \"kubernetes.io/projected/f9a2a7a5-468b-43f3-8528-1cc78b307324-kube-api-access-lfc98\") pod \"payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99\" (UID: \"f9a2a7a5-468b-43f3-8528-1cc78b307324\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99" Oct 30 22:04:02.010599 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:02.010562 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99" Oct 30 22:04:02.023235 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:02.023209 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:04:02.384401 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:02.384368 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99"] Oct 30 22:04:02.387047 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:04:02.387017 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a2a7a5_468b_43f3_8528_1cc78b307324.slice/crio-c4532d66baf7cb5451437cf08cc714378fe1ae70cce6d2b887b2b85fe833250c WatchSource:0}: Error finding container c4532d66baf7cb5451437cf08cc714378fe1ae70cce6d2b887b2b85fe833250c: Status 404 returned error can't find the container with id c4532d66baf7cb5451437cf08cc714378fe1ae70cce6d2b887b2b85fe833250c Oct 30 22:04:02.667778 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:02.667685 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99" event={"ID":"f9a2a7a5-468b-43f3-8528-1cc78b307324","Type":"ContainerStarted","Data":"c4532d66baf7cb5451437cf08cc714378fe1ae70cce6d2b887b2b85fe833250c"} Oct 30 22:04:04.252031 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:04.251992 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv"] Oct 30 22:04:04.261447 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:04.261412 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv" Oct 30 22:04:04.316071 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:04.316034 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv"] Oct 30 22:04:04.320666 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:04.320627 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de28863a-1f23-489f-aed6-9392cdf58442-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv\" (UID: \"de28863a-1f23-489f-aed6-9392cdf58442\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv" Oct 30 22:04:04.320863 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:04.320842 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxprq\" (UniqueName: \"kubernetes.io/projected/de28863a-1f23-489f-aed6-9392cdf58442-kube-api-access-pxprq\") pod \"payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv\" (UID: \"de28863a-1f23-489f-aed6-9392cdf58442\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv" Oct 30 22:04:04.356570 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:04.356534 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-7bb1c2\"" Oct 30 22:04:04.356570 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:04.356542 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-7bb1c2-dockercfg-6ts4j\"" Oct 30 22:04:04.398781 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:04.398738 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld"] Oct 30 22:04:04.416854 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:04.416821 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld" Oct 30 22:04:04.421870 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:04.421833 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxprq\" (UniqueName: \"kubernetes.io/projected/de28863a-1f23-489f-aed6-9392cdf58442-kube-api-access-pxprq\") pod \"payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv\" (UID: \"de28863a-1f23-489f-aed6-9392cdf58442\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv" Oct 30 22:04:04.422029 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:04.421925 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de28863a-1f23-489f-aed6-9392cdf58442-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv\" (UID: \"de28863a-1f23-489f-aed6-9392cdf58442\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv" Oct 30 22:04:04.422923 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:04.422897 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de28863a-1f23-489f-aed6-9392cdf58442-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv\" (UID: \"de28863a-1f23-489f-aed6-9392cdf58442\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv" Oct 30 22:04:04.456804 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:04.456766 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld"] Oct 30 22:04:04.481045 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:04.481009 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxprq\" (UniqueName: \"kubernetes.io/projected/de28863a-1f23-489f-aed6-9392cdf58442-kube-api-access-pxprq\") pod \"payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv\" (UID: \"de28863a-1f23-489f-aed6-9392cdf58442\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv" Oct 30 22:04:04.522914 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:04.522751 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2fg2\" (UniqueName: \"kubernetes.io/projected/9e5281f9-05da-4a88-87cd-4852a4edaba7-kube-api-access-k2fg2\") pod \"payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld\" (UID: \"9e5281f9-05da-4a88-87cd-4852a4edaba7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld" Oct 30 22:04:04.522914 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:04.522828 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5281f9-05da-4a88-87cd-4852a4edaba7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld\" (UID: \"9e5281f9-05da-4a88-87cd-4852a4edaba7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld" Oct 30 22:04:04.579282 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:04.579204 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv" Oct 30 22:04:04.624284 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:04.624222 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2fg2\" (UniqueName: \"kubernetes.io/projected/9e5281f9-05da-4a88-87cd-4852a4edaba7-kube-api-access-k2fg2\") pod \"payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld\" (UID: \"9e5281f9-05da-4a88-87cd-4852a4edaba7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld" Oct 30 22:04:04.624485 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:04.624333 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5281f9-05da-4a88-87cd-4852a4edaba7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld\" (UID: \"9e5281f9-05da-4a88-87cd-4852a4edaba7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld" Oct 30 22:04:04.625195 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:04.625167 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5281f9-05da-4a88-87cd-4852a4edaba7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld\" (UID: \"9e5281f9-05da-4a88-87cd-4852a4edaba7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld" Oct 30 22:04:04.684318 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:04.684285 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2fg2\" (UniqueName: \"kubernetes.io/projected/9e5281f9-05da-4a88-87cd-4852a4edaba7-kube-api-access-k2fg2\") pod \"payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld\" (UID: \"9e5281f9-05da-4a88-87cd-4852a4edaba7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld" Oct 30 22:04:04.736787 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:04.736312 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld" Oct 30 22:04:04.961078 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:04.961036 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv"] Oct 30 22:04:04.965685 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:04:04.965653 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde28863a_1f23_489f_aed6_9392cdf58442.slice/crio-e4c657b094d3f13df1482a5bf15c8e5bb3cb74757ec5545d544426c9591ba66b WatchSource:0}: Error finding container e4c657b094d3f13df1482a5bf15c8e5bb3cb74757ec5545d544426c9591ba66b: Status 404 returned error can't find the container with id e4c657b094d3f13df1482a5bf15c8e5bb3cb74757ec5545d544426c9591ba66b Oct 30 22:04:05.119449 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:05.119419 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld"] Oct 30 22:04:05.121961 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:04:05.121928 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5281f9_05da_4a88_87cd_4852a4edaba7.slice/crio-c45eb72a33693317d001cc8ed800264c39d81bbbd9eb2668bf1133f1474e2260 WatchSource:0}: Error finding container c45eb72a33693317d001cc8ed800264c39d81bbbd9eb2668bf1133f1474e2260: Status 404 returned error can't find the container with id c45eb72a33693317d001cc8ed800264c39d81bbbd9eb2668bf1133f1474e2260 Oct 30 22:04:05.685961 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:05.685921 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv" event={"ID":"de28863a-1f23-489f-aed6-9392cdf58442","Type":"ContainerStarted","Data":"e4c657b094d3f13df1482a5bf15c8e5bb3cb74757ec5545d544426c9591ba66b"} Oct 30 22:04:05.687141 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:05.687114 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld" event={"ID":"9e5281f9-05da-4a88-87cd-4852a4edaba7","Type":"ContainerStarted","Data":"c45eb72a33693317d001cc8ed800264c39d81bbbd9eb2668bf1133f1474e2260"} Oct 30 22:04:05.688687 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:05.688661 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99" event={"ID":"f9a2a7a5-468b-43f3-8528-1cc78b307324","Type":"ContainerStarted","Data":"ea914c779525256ba30f6a32d4a6345f873203fb025f8c1e655dcc2e36fd40e4"} Oct 30 22:04:05.688811 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:05.688805 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99" Oct 30 22:04:05.804713 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:05.804660 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99" podStartSLOduration=1.945485337 podStartE2EDuration="4.80464547s" totalImagesPullingTime="2.859160133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:05.803316098 +0000 UTC m=+848.355492396" watchObservedRunningTime="2026-10-30 22:04:05.80464547 +0000 UTC m=+848.356821769" Oct 30 22:04:06.698883 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:06.698845 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv" event={"ID":"de28863a-1f23-489f-aed6-9392cdf58442","Type":"ContainerStarted","Data":"303888e59a3a9f1a73bd7cfd11fdee36ee7d32f754e4abeb2439594ff38dee2b"} Oct 30 22:04:06.699521 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:06.699227 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv" Oct 30 22:04:06.701793 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:06.701573 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld" event={"ID":"9e5281f9-05da-4a88-87cd-4852a4edaba7","Type":"ContainerStarted","Data":"01c1bab04f84beb052aadcb6c5309c91370e91c22348dec2387e92aab11a8069"} Oct 30 22:04:06.701955 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:06.701902 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld" Oct 30 22:04:06.811520 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:06.811465 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv" podStartSLOduration=1.859459339 podStartE2EDuration="2.811449797s" totalImagesPullingTime="951.990458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:06.809650376 +0000 UTC m=+849.361826677" watchObservedRunningTime="2026-10-30 22:04:06.811449797 +0000 UTC m=+849.363626096" Oct 30 22:04:06.921892 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:06.921836 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld" podStartSLOduration=2.068267051 podStartE2EDuration="2.921820153s" totalImagesPullingTime="853.553102ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:06.920738678 +0000 UTC m=+849.472914979" watchObservedRunningTime="2026-10-30 22:04:06.921820153 +0000 UTC m=+849.473996478" Oct 30 22:04:07.146725 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:07.146683 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2"] Oct 30 22:04:07.151474 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:07.151447 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2" Oct 30 22:04:07.204843 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:07.204809 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-65e59f\"" Oct 30 22:04:07.205017 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:07.204927 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-65e59f-dockercfg-pgbkn\"" Oct 30 22:04:07.206826 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:07.206795 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2"] Oct 30 22:04:07.252201 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:07.252164 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27wrv\" (UniqueName: \"kubernetes.io/projected/9a510562-a7e8-4abe-963e-3ddd4ff88162-kube-api-access-27wrv\") pod \"payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2\" (UID: \"9a510562-a7e8-4abe-963e-3ddd4ff88162\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2" Oct 30 22:04:07.252393 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:07.252354 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a510562-a7e8-4abe-963e-3ddd4ff88162-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2\" (UID: \"9a510562-a7e8-4abe-963e-3ddd4ff88162\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2" Oct 30 22:04:07.353082 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:07.353047 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27wrv\" (UniqueName: \"kubernetes.io/projected/9a510562-a7e8-4abe-963e-3ddd4ff88162-kube-api-access-27wrv\") pod \"payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2\" (UID: \"9a510562-a7e8-4abe-963e-3ddd4ff88162\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2" Oct 30 22:04:07.353350 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:07.353139 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a510562-a7e8-4abe-963e-3ddd4ff88162-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2\" (UID: \"9a510562-a7e8-4abe-963e-3ddd4ff88162\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2" Oct 30 22:04:07.353882 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:07.353862 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a510562-a7e8-4abe-963e-3ddd4ff88162-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2\" (UID: \"9a510562-a7e8-4abe-963e-3ddd4ff88162\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2" Oct 30 22:04:07.412736 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:07.412654 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27wrv\" (UniqueName: \"kubernetes.io/projected/9a510562-a7e8-4abe-963e-3ddd4ff88162-kube-api-access-27wrv\") pod \"payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2\" (UID: \"9a510562-a7e8-4abe-963e-3ddd4ff88162\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2" Oct 30 22:04:07.465232 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:07.465193 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2" Oct 30 22:04:07.835326 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:07.835070 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2"] Oct 30 22:04:07.837801 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:04:07.837774 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a510562_a7e8_4abe_963e_3ddd4ff88162.slice/crio-6e196f9ea9c4dd2a8b1f79c8a7d09419ccd9e2298fa02473e4e6c1f591eeef48 WatchSource:0}: Error finding container 6e196f9ea9c4dd2a8b1f79c8a7d09419ccd9e2298fa02473e4e6c1f591eeef48: Status 404 returned error can't find the container with id 6e196f9ea9c4dd2a8b1f79c8a7d09419ccd9e2298fa02473e4e6c1f591eeef48 Oct 30 22:04:08.712588 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:08.712550 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2" event={"ID":"9a510562-a7e8-4abe-963e-3ddd4ff88162","Type":"ContainerStarted","Data":"f435bc68dd004f99ef69ad5739aca3634b4343eff946ebef1231e5ecffa6c2d9"} Oct 30 22:04:08.712773 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:08.712612 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2" event={"ID":"9a510562-a7e8-4abe-963e-3ddd4ff88162","Type":"ContainerStarted","Data":"6e196f9ea9c4dd2a8b1f79c8a7d09419ccd9e2298fa02473e4e6c1f591eeef48"} Oct 30 22:04:08.712773 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:08.712745 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2" Oct 30 22:04:08.825002 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:08.824939 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2" podStartSLOduration=1.824919252 podStartE2EDuration="1.824919252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:08.823407471 +0000 UTC m=+851.375583772" watchObservedRunningTime="2026-10-30 22:04:08.824919252 +0000 UTC m=+851.377095562" Oct 30 22:04:09.811313 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:09.811277 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw"] Oct 30 22:04:09.815339 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:09.815315 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw" Oct 30 22:04:09.873127 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:09.873095 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-fb0b93\"" Oct 30 22:04:09.877646 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:09.877607 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hd84\" (UniqueName: \"kubernetes.io/projected/a35ddaa8-6863-4c1f-b9e8-711a8e76e006-kube-api-access-4hd84\") pod \"payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw\" (UID: \"a35ddaa8-6863-4c1f-b9e8-711a8e76e006\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw" Oct 30 22:04:09.877816 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:09.877742 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a35ddaa8-6863-4c1f-b9e8-711a8e76e006-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw\" (UID: \"a35ddaa8-6863-4c1f-b9e8-711a8e76e006\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw" Oct 30 22:04:09.912886 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:09.912843 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-fb0b93-dockercfg-fdjjh\"" Oct 30 22:04:09.914444 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:09.914419 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw"] Oct 30 22:04:09.978574 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:09.978540 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hd84\" (UniqueName: \"kubernetes.io/projected/a35ddaa8-6863-4c1f-b9e8-711a8e76e006-kube-api-access-4hd84\") pod \"payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw\" (UID: \"a35ddaa8-6863-4c1f-b9e8-711a8e76e006\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw" Oct 30 22:04:09.978762 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:09.978601 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a35ddaa8-6863-4c1f-b9e8-711a8e76e006-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw\" (UID: \"a35ddaa8-6863-4c1f-b9e8-711a8e76e006\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw" Oct 30 22:04:09.979373 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:09.979354 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a35ddaa8-6863-4c1f-b9e8-711a8e76e006-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw\" (UID: \"a35ddaa8-6863-4c1f-b9e8-711a8e76e006\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw" Oct 30 22:04:10.037606 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:10.037568 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hd84\" (UniqueName: \"kubernetes.io/projected/a35ddaa8-6863-4c1f-b9e8-711a8e76e006-kube-api-access-4hd84\") pod \"payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw\" (UID: \"a35ddaa8-6863-4c1f-b9e8-711a8e76e006\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw" Oct 30 22:04:10.129658 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:10.129535 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw" Oct 30 22:04:10.503919 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:10.503892 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw"] Oct 30 22:04:10.506606 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:04:10.506572 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda35ddaa8_6863_4c1f_b9e8_711a8e76e006.slice/crio-6b52d18cbd5930ca22fc4bbcfa426357ad19fec5574e688808b0cc3c2c9f528d WatchSource:0}: Error finding container 6b52d18cbd5930ca22fc4bbcfa426357ad19fec5574e688808b0cc3c2c9f528d: Status 404 returned error can't find the container with id 6b52d18cbd5930ca22fc4bbcfa426357ad19fec5574e688808b0cc3c2c9f528d Oct 30 22:04:10.726141 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:10.726107 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw" event={"ID":"a35ddaa8-6863-4c1f-b9e8-711a8e76e006","Type":"ContainerStarted","Data":"cfcb71a25305fcc666a6a8f5d539f64d19b184b4678358ff364655ca1e23cbb5"} Oct 30 22:04:10.726337 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:10.726173 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw" Oct 30 22:04:10.726337 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:10.726200 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw" event={"ID":"a35ddaa8-6863-4c1f-b9e8-711a8e76e006","Type":"ContainerStarted","Data":"6b52d18cbd5930ca22fc4bbcfa426357ad19fec5574e688808b0cc3c2c9f528d"} Oct 30 22:04:10.840523 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:10.840450 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw" podStartSLOduration=1.840426149 podStartE2EDuration="1.840426149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:10.839478851 +0000 UTC m=+853.391655152" watchObservedRunningTime="2026-10-30 22:04:10.840426149 +0000 UTC m=+853.392602452" Oct 30 22:04:11.064224 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:11.064177 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:04:16.705676 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:16.705647 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99" Oct 30 22:04:17.709527 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:17.709493 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv" Oct 30 22:04:17.709998 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:17.709565 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld" Oct 30 22:04:19.721368 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:19.721339 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2" Oct 30 22:04:21.734390 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:21.734356 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw" Oct 30 22:04:27.909997 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:04:27.909950 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:04:28.317226 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:28.317184 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:04:29.184578 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.184545 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4"] Oct 30 22:04:29.189621 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.189601 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.239468 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.239431 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a00d9b-openshift-default-dockercfg-w7t4n\"" Oct 30 22:04:29.263968 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.263919 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-envoy\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.264156 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.263989 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88f78b1a-a774-44f1-8a64-8786dcdcf528-istiod-ca-cert\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.264156 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.264065 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-credential-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.264156 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.264091 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-token\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.264156 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.264141 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-podinfo\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.264401 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.264219 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-data\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.264401 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.264265 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-workload-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.264401 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.264291 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjhpn\" (UniqueName: \"kubernetes.io/projected/88f78b1a-a774-44f1-8a64-8786dcdcf528-kube-api-access-gjhpn\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.264401 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.264325 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-workload-certs\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.287867 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.287827 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4"] Oct 30 22:04:29.365114 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.365079 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-credential-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.365114 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.365119 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-token\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.365441 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.365158 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-podinfo\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.365441 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.365210 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-data\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.365441 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.365233 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-workload-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.365441 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.365289 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjhpn\" (UniqueName: \"kubernetes.io/projected/88f78b1a-a774-44f1-8a64-8786dcdcf528-kube-api-access-gjhpn\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.365441 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.365322 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-workload-certs\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.365441 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.365366 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-envoy\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.365441 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.365407 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88f78b1a-a774-44f1-8a64-8786dcdcf528-istiod-ca-cert\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.365810 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.365587 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-credential-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.365810 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.365667 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-data\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.365810 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.365684 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-workload-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.365810 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.365769 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-workload-certs\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.366849 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.366822 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88f78b1a-a774-44f1-8a64-8786dcdcf528-istiod-ca-cert\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.368098 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.368074 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-podinfo\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.368208 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.368153 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-envoy\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.424098 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.424059 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-token\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.424278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.424212 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjhpn\" (UniqueName: \"kubernetes.io/projected/88f78b1a-a774-44f1-8a64-8786dcdcf528-kube-api-access-gjhpn\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.503722 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.503630 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:29.926946 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.926897 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4"] Oct 30 22:04:29.961013 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:04:29.960979 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f78b1a_a774_44f1_8a64_8786dcdcf528.slice/crio-64053369c6fa22cdfe07d4bf392b5a7a480fd158364284440c70155d67203248 WatchSource:0}: Error finding container 64053369c6fa22cdfe07d4bf392b5a7a480fd158364284440c70155d67203248: Status 404 returned error can't find the container with id 64053369c6fa22cdfe07d4bf392b5a7a480fd158364284440c70155d67203248 Oct 30 22:04:29.963434 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.963402 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:04:29.963506 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.963469 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:04:29.963506 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:29.963501 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:04:30.839719 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:30.839678 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" event={"ID":"88f78b1a-a774-44f1-8a64-8786dcdcf528","Type":"ContainerStarted","Data":"47eee70ff05a9a50b15660f5d6609211bd00f3e32e1344a24e7891893dc87462"} Oct 30 22:04:30.840131 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:30.839758 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" event={"ID":"88f78b1a-a774-44f1-8a64-8786dcdcf528","Type":"ContainerStarted","Data":"64053369c6fa22cdfe07d4bf392b5a7a480fd158364284440c70155d67203248"} Oct 30 22:04:31.161087 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:31.160985 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" podStartSLOduration=2.160966661 podStartE2EDuration="2.160966661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:31.016407509 +0000 UTC m=+873.568583810" watchObservedRunningTime="2026-10-30 22:04:31.160966661 +0000 UTC m=+873.713142955" Oct 30 22:04:31.161857 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:31.161831 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8"] Oct 30 22:04:31.165847 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:31.165825 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8" Oct 30 22:04:31.224289 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:31.223891 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 22:04:31.224289 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:31.224019 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jglqk\"" Oct 30 22:04:31.224289 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:31.224084 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:04:31.226454 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:31.226058 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:04:31.226454 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:31.226130 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-c9dbe6-serving-cert\"" Oct 30 22:04:31.226454 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:31.226156 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8"] Oct 30 22:04:31.226865 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:31.226837 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8\" (UID: \"2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8" Oct 30 22:04:31.227030 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:31.227009 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79t6n\" (UniqueName: \"kubernetes.io/projected/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-kube-api-access-79t6n\") pod \"maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8\" (UID: \"2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8" Oct 30 22:04:31.227181 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:31.227162 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-maas-api-tls\") pod \"maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8\" (UID: \"2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8" Oct 30 22:04:31.227374 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:31.227342 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-metrics-certs\") pod \"maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8\" (UID: \"2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8" Oct 30 22:04:31.331278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:31.329855 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8\" (UID: \"2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8" Oct 30 22:04:31.331278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:31.329912 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79t6n\" (UniqueName: \"kubernetes.io/projected/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-kube-api-access-79t6n\") pod \"maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8\" (UID: \"2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8" Oct 30 22:04:31.331278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:31.329952 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-maas-api-tls\") pod \"maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8\" (UID: \"2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8" Oct 30 22:04:31.331278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:31.329989 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-metrics-certs\") pod \"maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8\" (UID: \"2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8" Oct 30 22:04:31.334695 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:04:31.334082 6006 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-serving-cert: secret "maas-api-e2e-worker-w5-c9dbe6-serving-cert" not found Oct 30 22:04:31.334695 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:04:31.334171 6006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-maas-api-tls podName:2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f nodeName:}" failed. No retries permitted until 2026-10-30 22:04:31.83414822 +0000 UTC m=+874.386324521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-maas-api-tls") pod "maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8" (UID: "2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f") : secret "maas-api-e2e-worker-w5-c9dbe6-serving-cert" not found Oct 30 22:04:31.335681 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:31.335635 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8\" (UID: \"2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8" Oct 30 22:04:31.339913 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:31.339867 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-metrics-certs\") pod \"maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8\" (UID: \"2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8" Oct 30 22:04:31.394723 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:31.394686 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79t6n\" (UniqueName: \"kubernetes.io/projected/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-kube-api-access-79t6n\") pod \"maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8\" (UID: \"2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8" Oct 30 22:04:31.523469 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:31.523369 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:31.523621 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:31.523435 6006 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 22:04:31.523688 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:31.523627 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" podUID="88f78b1a-a774-44f1-8a64-8786dcdcf528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 22:04:31.835473 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:31.835424 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-maas-api-tls\") pod \"maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8\" (UID: \"2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8" Oct 30 22:04:31.838080 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:31.838044 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-maas-api-tls\") pod \"maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8\" (UID: \"2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8" Oct 30 22:04:32.080313 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:32.080277 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8" Oct 30 22:04:32.484487 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:32.484450 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8"] Oct 30 22:04:32.487800 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:04:32.487765 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bfa3a54_fb74_4985_85c2_df5d7fb3ed1f.slice/crio-ab9975f82faac96ff5995164e88ca4ad0c4ef690f0b106e6cef481a84d4a5ec7 WatchSource:0}: Error finding container ab9975f82faac96ff5995164e88ca4ad0c4ef690f0b106e6cef481a84d4a5ec7: Status 404 returned error can't find the container with id ab9975f82faac96ff5995164e88ca4ad0c4ef690f0b106e6cef481a84d4a5ec7 Oct 30 22:04:32.511275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:32.507698 6006 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 22:04:32.511275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:32.507767 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" podUID="88f78b1a-a774-44f1-8a64-8786dcdcf528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 22:04:32.851186 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:32.851146 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8" event={"ID":"2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f","Type":"ContainerStarted","Data":"ab9975f82faac96ff5995164e88ca4ad0c4ef690f0b106e6cef481a84d4a5ec7"} Oct 30 22:04:33.523853 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:33.523807 6006 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 22:04:33.524334 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:33.523885 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" podUID="88f78b1a-a774-44f1-8a64-8786dcdcf528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 22:04:34.522137 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:34.521679 6006 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 22:04:34.522137 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:34.521744 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" podUID="88f78b1a-a774-44f1-8a64-8786dcdcf528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 22:04:35.523276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:35.522648 6006 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 22:04:35.523276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:35.522713 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" podUID="88f78b1a-a774-44f1-8a64-8786dcdcf528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 22:04:36.508454 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:36.508422 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:36.508849 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:36.508829 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:36.509553 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:36.509537 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:04:36.883189 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:36.883148 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8" event={"ID":"2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f","Type":"ContainerStarted","Data":"c7f2b6691b55b08111662b3ba960ca134218f4fb6ab91e0f26316a7f604f4b4c"} Oct 30 22:04:36.883689 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:36.883388 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8" Oct 30 22:04:36.996398 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:36.996334 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8" podStartSLOduration=2.180218518 podStartE2EDuration="5.996316131s" totalImagesPullingTime="3.816097613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:36.995830308 +0000 UTC m=+879.548006608" watchObservedRunningTime="2026-10-30 22:04:36.996316131 +0000 UTC m=+879.548492433" Oct 30 22:04:39.043225 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:39.043185 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:04:42.894149 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:42.894116 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8" Oct 30 22:04:48.102213 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.102175 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569"] Oct 30 22:04:48.106074 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.106053 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:04:48.156761 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.156720 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-bc714d\"/\"openshift-service-ca.crt\"" Oct 30 22:04:48.203423 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.203384 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-bc714d\"/\"kube-root-ca.crt\"" Oct 30 22:04:48.203994 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.203953 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-bc714d\"/\"simulator-w2-bc714d-kserve-self-signed-certs\"" Oct 30 22:04:48.208037 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.207951 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569"] Oct 30 22:04:48.212777 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.203969 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-bc714d\"/\"default-dockercfg-hzf47\"" Oct 30 22:04:48.213681 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.213649 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm2m5\" (UniqueName: \"kubernetes.io/projected/fe28bccf-50cc-4c80-b29d-47810d3aabd2-kube-api-access-bm2m5\") pod \"simulator-w2-bc714d-kserve-cc58c74bd-68569\" (UID: \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\") " pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:04:48.213816 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.213699 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-dshm\") pod \"simulator-w2-bc714d-kserve-cc58c74bd-68569\" (UID: \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\") " pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:04:48.213878 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.213828 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-home\") pod \"simulator-w2-bc714d-kserve-cc58c74bd-68569\" (UID: \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\") " pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:04:48.213929 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.213885 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-model-cache\") pod \"simulator-w2-bc714d-kserve-cc58c74bd-68569\" (UID: \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\") " pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:04:48.213986 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.213960 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe28bccf-50cc-4c80-b29d-47810d3aabd2-tls-certs\") pod \"simulator-w2-bc714d-kserve-cc58c74bd-68569\" (UID: \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\") " pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:04:48.214034 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.214015 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-tmp-dir\") pod \"simulator-w2-bc714d-kserve-cc58c74bd-68569\" (UID: \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\") " pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:04:48.315320 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.315279 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-dshm\") pod \"simulator-w2-bc714d-kserve-cc58c74bd-68569\" (UID: \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\") " pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:04:48.315528 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.315342 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-home\") pod \"simulator-w2-bc714d-kserve-cc58c74bd-68569\" (UID: \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\") " pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:04:48.315528 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.315375 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-model-cache\") pod \"simulator-w2-bc714d-kserve-cc58c74bd-68569\" (UID: \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\") " pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:04:48.315528 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.315417 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe28bccf-50cc-4c80-b29d-47810d3aabd2-tls-certs\") pod \"simulator-w2-bc714d-kserve-cc58c74bd-68569\" (UID: \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\") " pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:04:48.315528 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.315458 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-tmp-dir\") pod \"simulator-w2-bc714d-kserve-cc58c74bd-68569\" (UID: \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\") " pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:04:48.315528 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.315526 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm2m5\" (UniqueName: \"kubernetes.io/projected/fe28bccf-50cc-4c80-b29d-47810d3aabd2-kube-api-access-bm2m5\") pod \"simulator-w2-bc714d-kserve-cc58c74bd-68569\" (UID: \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\") " pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:04:48.315857 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.315829 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-home\") pod \"simulator-w2-bc714d-kserve-cc58c74bd-68569\" (UID: \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\") " pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:04:48.315921 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.315849 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-model-cache\") pod \"simulator-w2-bc714d-kserve-cc58c74bd-68569\" (UID: \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\") " pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:04:48.316094 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.316072 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-tmp-dir\") pod \"simulator-w2-bc714d-kserve-cc58c74bd-68569\" (UID: \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\") " pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:04:48.317611 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.317586 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-dshm\") pod \"simulator-w2-bc714d-kserve-cc58c74bd-68569\" (UID: \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\") " pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:04:48.318484 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.318457 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe28bccf-50cc-4c80-b29d-47810d3aabd2-tls-certs\") pod \"simulator-w2-bc714d-kserve-cc58c74bd-68569\" (UID: \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\") " pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:04:48.374816 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.374731 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm2m5\" (UniqueName: \"kubernetes.io/projected/fe28bccf-50cc-4c80-b29d-47810d3aabd2-kube-api-access-bm2m5\") pod \"simulator-w2-bc714d-kserve-cc58c74bd-68569\" (UID: \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\") " pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:04:48.418717 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.418673 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:04:48.810658 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.810633 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569"] Oct 30 22:04:48.812229 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:04:48.812195 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe28bccf_50cc_4c80_b29d_47810d3aabd2.slice/crio-a6473218f6a74d88b674fa71888399b55ae9ada188d24e8ceb121463fded629e WatchSource:0}: Error finding container a6473218f6a74d88b674fa71888399b55ae9ada188d24e8ceb121463fded629e: Status 404 returned error can't find the container with id a6473218f6a74d88b674fa71888399b55ae9ada188d24e8ceb121463fded629e Oct 30 22:04:48.953704 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:48.953670 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" event={"ID":"fe28bccf-50cc-4c80-b29d-47810d3aabd2","Type":"ContainerStarted","Data":"a6473218f6a74d88b674fa71888399b55ae9ada188d24e8ceb121463fded629e"} Oct 30 22:04:50.967388 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:50.967344 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" event={"ID":"fe28bccf-50cc-4c80-b29d-47810d3aabd2","Type":"ContainerStarted","Data":"a6ed2634bf8a97594ab7a9ad2bf794a439ba2787985c34091decf53214b42034"} Oct 30 22:04:51.080675 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:51.080623 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" podStartSLOduration=1.443762467 podStartE2EDuration="3.080608081s" totalImagesPullingTime="1.636845614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:51.078519473 +0000 UTC m=+893.630695773" watchObservedRunningTime="2026-10-30 22:04:51.080608081 +0000 UTC m=+893.632784381" Oct 30 22:04:51.419052 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:51.419009 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:04:55.515838 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:55.515799 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:04:57.698837 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:57.698804 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 22:04:57.699614 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:57.699594 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 22:04:57.913977 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:04:57.913931 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:04:58.419739 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:58.419705 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:04:58.433033 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:58.433002 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:04:59.023743 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:04:59.023704 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:05:09.343661 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:09.343622 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp"] Oct 30 22:05:09.350728 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:09.350706 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:05:09.431337 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:09.431302 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-dshm\") pod \"simulator-w0-779f32-kserve-85c494d775-d4bkp\" (UID: \"8040ac93-043b-49a2-8388-f44f1f93e440\") " pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:05:09.431337 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:09.431339 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-model-cache\") pod \"simulator-w0-779f32-kserve-85c494d775-d4bkp\" (UID: \"8040ac93-043b-49a2-8388-f44f1f93e440\") " pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:05:09.431544 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:09.431372 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8040ac93-043b-49a2-8388-f44f1f93e440-tls-certs\") pod \"simulator-w0-779f32-kserve-85c494d775-d4bkp\" (UID: \"8040ac93-043b-49a2-8388-f44f1f93e440\") " pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:05:09.431588 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:09.431533 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-tmp-dir\") pod \"simulator-w0-779f32-kserve-85c494d775-d4bkp\" (UID: \"8040ac93-043b-49a2-8388-f44f1f93e440\") " pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:05:09.431588 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:09.431574 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8sjv\" (UniqueName: \"kubernetes.io/projected/8040ac93-043b-49a2-8388-f44f1f93e440-kube-api-access-z8sjv\") pod \"simulator-w0-779f32-kserve-85c494d775-d4bkp\" (UID: \"8040ac93-043b-49a2-8388-f44f1f93e440\") " pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:05:09.431660 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:09.431638 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-home\") pod \"simulator-w0-779f32-kserve-85c494d775-d4bkp\" (UID: \"8040ac93-043b-49a2-8388-f44f1f93e440\") " pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:05:09.449193 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:09.449152 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-779f32\"/\"openshift-service-ca.crt\"" Oct 30 22:05:09.449193 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:09.449159 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-779f32\"/\"default-dockercfg-lcntz\"" Oct 30 22:05:09.449406 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:09.449378 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-779f32\"/\"simulator-w0-779f32-kserve-self-signed-certs\"" Oct 30 22:05:09.449448 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:09.449394 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-779f32\"/\"kube-root-ca.crt\"" Oct 30 22:05:09.450994 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:09.450974 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp"] Oct 30 22:05:09.533162 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:09.533117 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-tmp-dir\") pod \"simulator-w0-779f32-kserve-85c494d775-d4bkp\" (UID: \"8040ac93-043b-49a2-8388-f44f1f93e440\") " pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:05:09.533162 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:09.533165 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8sjv\" (UniqueName: \"kubernetes.io/projected/8040ac93-043b-49a2-8388-f44f1f93e440-kube-api-access-z8sjv\") pod \"simulator-w0-779f32-kserve-85c494d775-d4bkp\" (UID: \"8040ac93-043b-49a2-8388-f44f1f93e440\") " pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:05:09.533483 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:09.533204 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-home\") pod \"simulator-w0-779f32-kserve-85c494d775-d4bkp\" (UID: \"8040ac93-043b-49a2-8388-f44f1f93e440\") " pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:05:09.533483 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:09.533235 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-dshm\") pod \"simulator-w0-779f32-kserve-85c494d775-d4bkp\" (UID: \"8040ac93-043b-49a2-8388-f44f1f93e440\") " pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:05:09.533483 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:09.533268 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-model-cache\") pod \"simulator-w0-779f32-kserve-85c494d775-d4bkp\" (UID: \"8040ac93-043b-49a2-8388-f44f1f93e440\") " pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:05:09.533483 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:09.533354 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8040ac93-043b-49a2-8388-f44f1f93e440-tls-certs\") pod \"simulator-w0-779f32-kserve-85c494d775-d4bkp\" (UID: \"8040ac93-043b-49a2-8388-f44f1f93e440\") " pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:05:09.533761 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:09.533652 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-tmp-dir\") pod \"simulator-w0-779f32-kserve-85c494d775-d4bkp\" (UID: \"8040ac93-043b-49a2-8388-f44f1f93e440\") " pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:05:09.533761 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:09.533710 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-home\") pod \"simulator-w0-779f32-kserve-85c494d775-d4bkp\" (UID: \"8040ac93-043b-49a2-8388-f44f1f93e440\") " pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:05:09.533761 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:09.533738 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-model-cache\") pod \"simulator-w0-779f32-kserve-85c494d775-d4bkp\" (UID: \"8040ac93-043b-49a2-8388-f44f1f93e440\") " pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:05:09.535803 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:09.535777 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-dshm\") pod \"simulator-w0-779f32-kserve-85c494d775-d4bkp\" (UID: \"8040ac93-043b-49a2-8388-f44f1f93e440\") " pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:05:09.536193 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:09.536172 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8040ac93-043b-49a2-8388-f44f1f93e440-tls-certs\") pod \"simulator-w0-779f32-kserve-85c494d775-d4bkp\" (UID: \"8040ac93-043b-49a2-8388-f44f1f93e440\") " pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:05:09.592442 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:09.592405 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8sjv\" (UniqueName: \"kubernetes.io/projected/8040ac93-043b-49a2-8388-f44f1f93e440-kube-api-access-z8sjv\") pod \"simulator-w0-779f32-kserve-85c494d775-d4bkp\" (UID: \"8040ac93-043b-49a2-8388-f44f1f93e440\") " pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:05:09.664947 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:09.664856 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:05:10.052622 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:10.052577 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp"] Oct 30 22:05:10.055566 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:05:10.055537 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8040ac93_043b_49a2_8388_f44f1f93e440.slice/crio-2b8918ed4e1590126c8aecc8ee84adb9a0e892be98769070369dbd061a62635f WatchSource:0}: Error finding container 2b8918ed4e1590126c8aecc8ee84adb9a0e892be98769070369dbd061a62635f: Status 404 returned error can't find the container with id 2b8918ed4e1590126c8aecc8ee84adb9a0e892be98769070369dbd061a62635f Oct 30 22:05:10.075892 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:10.075862 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" event={"ID":"8040ac93-043b-49a2-8388-f44f1f93e440","Type":"ContainerStarted","Data":"2b8918ed4e1590126c8aecc8ee84adb9a0e892be98769070369dbd061a62635f"} Oct 30 22:05:11.082539 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:11.082499 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" event={"ID":"8040ac93-043b-49a2-8388-f44f1f93e440","Type":"ContainerStarted","Data":"53c7f3cd8d30cb1f2d26502be354bfef9fa843605a88e51f37b396a639512917"} Oct 30 22:05:11.196516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:11.196456 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" podStartSLOduration=2.196438791 podStartE2EDuration="2.196438791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:11.194538621 +0000 UTC m=+913.746714922" watchObservedRunningTime="2026-10-30 22:05:11.196438791 +0000 UTC m=+913.748615092" Oct 30 22:05:11.665881 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:11.665846 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:05:14.974469 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:14.974424 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:05:19.665909 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:19.665869 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:05:19.678792 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:19.678764 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:05:20.154807 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:20.154775 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:05:27.449692 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:27.449653 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn"] Oct 30 22:05:27.456932 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:27.456908 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:05:27.522646 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:27.522603 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-tmp-dir\") pod \"simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn\" (UID: \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\") " pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:05:27.522838 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:27.522747 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c71fda3-adf4-4b08-8a27-0065ac056a1d-tls-certs\") pod \"simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn\" (UID: \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\") " pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:05:27.522838 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:27.522814 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-model-cache\") pod \"simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn\" (UID: \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\") " pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:05:27.522976 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:27.522894 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmk98\" (UniqueName: \"kubernetes.io/projected/8c71fda3-adf4-4b08-8a27-0065ac056a1d-kube-api-access-vmk98\") pod \"simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn\" (UID: \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\") " pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:05:27.522976 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:27.522929 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-home\") pod \"simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn\" (UID: \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\") " pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:05:27.522976 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:27.522954 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-dshm\") pod \"simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn\" (UID: \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\") " pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:05:27.524063 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:27.524039 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a00d9b\"/\"openshift-service-ca.crt\"" Oct 30 22:05:27.524176 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:27.524115 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a00d9b\"/\"kube-root-ca.crt\"" Oct 30 22:05:27.525949 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:27.525924 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn"] Oct 30 22:05:27.553484 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:27.553451 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a00d9b\"/\"default-dockercfg-ttmlk\"" Oct 30 22:05:27.553630 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:27.553506 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a00d9b\"/\"simulator-w1-a00d9b-kserve-self-signed-certs\"" Oct 30 22:05:27.624033 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:27.623999 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-tmp-dir\") pod \"simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn\" (UID: \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\") " pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:05:27.624220 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:27.624051 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c71fda3-adf4-4b08-8a27-0065ac056a1d-tls-certs\") pod \"simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn\" (UID: \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\") " pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:05:27.624220 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:27.624091 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-model-cache\") pod \"simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn\" (UID: \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\") " pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:05:27.624220 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:27.624137 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmk98\" (UniqueName: \"kubernetes.io/projected/8c71fda3-adf4-4b08-8a27-0065ac056a1d-kube-api-access-vmk98\") pod \"simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn\" (UID: \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\") " pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:05:27.624220 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:27.624158 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-home\") pod \"simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn\" (UID: \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\") " pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:05:27.624220 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:27.624177 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-dshm\") pod \"simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn\" (UID: \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\") " pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:05:27.624516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:27.624483 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-tmp-dir\") pod \"simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn\" (UID: \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\") " pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:05:27.624580 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:27.624563 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-model-cache\") pod \"simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn\" (UID: \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\") " pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:05:27.624666 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:27.624640 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-home\") pod \"simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn\" (UID: \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\") " pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:05:27.626384 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:27.626366 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-dshm\") pod \"simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn\" (UID: \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\") " pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:05:27.626792 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:27.626772 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c71fda3-adf4-4b08-8a27-0065ac056a1d-tls-certs\") pod \"simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn\" (UID: \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\") " pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:05:27.683161 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:27.683124 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmk98\" (UniqueName: \"kubernetes.io/projected/8c71fda3-adf4-4b08-8a27-0065ac056a1d-kube-api-access-vmk98\") pod \"simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn\" (UID: \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\") " pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:05:27.771183 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:27.771082 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:05:27.921448 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:05:27.921405 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:05:28.153445 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:28.153409 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn"] Oct 30 22:05:28.155617 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:05:28.155583 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c71fda3_adf4_4b08_8a27_0065ac056a1d.slice/crio-a3956a936c8b7b0603af8c109c7b8c3f086553b549dc6502eef3b77add552c06 WatchSource:0}: Error finding container a3956a936c8b7b0603af8c109c7b8c3f086553b549dc6502eef3b77add552c06: Status 404 returned error can't find the container with id a3956a936c8b7b0603af8c109c7b8c3f086553b549dc6502eef3b77add552c06 Oct 30 22:05:28.184486 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:28.184445 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" event={"ID":"8c71fda3-adf4-4b08-8a27-0065ac056a1d","Type":"ContainerStarted","Data":"a3956a936c8b7b0603af8c109c7b8c3f086553b549dc6502eef3b77add552c06"} Oct 30 22:05:29.191603 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:29.191567 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" event={"ID":"8c71fda3-adf4-4b08-8a27-0065ac056a1d","Type":"ContainerStarted","Data":"3f9f2b4ae0b8cee22c4c7b599a6def74ca28b91c76c68bcb6e74a1107238c824"} Oct 30 22:05:29.771685 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:29.771655 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:05:34.991400 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:34.991340 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podStartSLOduration=7.991323602 podStartE2EDuration="7.991323602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:29.304177831 +0000 UTC m=+931.856354130" watchObservedRunningTime="2026-10-30 22:05:34.991323602 +0000 UTC m=+937.543499901" Oct 30 22:05:34.992582 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:34.992557 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:05:37.771339 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:37.771295 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:05:37.784798 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:37.784760 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:05:38.253327 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:38.253296 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:05:50.348229 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:05:50.348182 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:05:57.912295 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:05:57.912213 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:06:01.021431 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.021389 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss"] Oct 30 22:06:01.033566 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.033536 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:06:01.073013 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.072972 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-home\") pod \"simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss\" (UID: \"49f28c58-bbe2-4110-8ac1-42e629c8be80\") " pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:06:01.073289 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.073266 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49f28c58-bbe2-4110-8ac1-42e629c8be80-tls-certs\") pod \"simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss\" (UID: \"49f28c58-bbe2-4110-8ac1-42e629c8be80\") " pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:06:01.073421 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.073405 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gv62\" (UniqueName: \"kubernetes.io/projected/49f28c58-bbe2-4110-8ac1-42e629c8be80-kube-api-access-7gv62\") pod \"simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss\" (UID: \"49f28c58-bbe2-4110-8ac1-42e629c8be80\") " pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:06:01.073618 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.073601 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-tmp-dir\") pod \"simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss\" (UID: \"49f28c58-bbe2-4110-8ac1-42e629c8be80\") " pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:06:01.073736 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.073721 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-model-cache\") pod \"simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss\" (UID: \"49f28c58-bbe2-4110-8ac1-42e629c8be80\") " pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:06:01.073899 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.073885 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-dshm\") pod \"simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss\" (UID: \"49f28c58-bbe2-4110-8ac1-42e629c8be80\") " pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:06:01.143717 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.143519 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss"] Oct 30 22:06:01.174712 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.165706 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-c9dbe6\"/\"kube-root-ca.crt\"" Oct 30 22:06:01.174712 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.165834 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c9dbe6\"/\"default-dockercfg-fhdlt\"" Oct 30 22:06:01.174712 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.165894 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-c9dbe6\"/\"openshift-service-ca.crt\"" Oct 30 22:06:01.174712 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.166009 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c9dbe6\"/\"simulator-w5-c9dbe6-kserve-self-signed-certs\"" Oct 30 22:06:01.175850 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.175817 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-dshm\") pod \"simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss\" (UID: \"49f28c58-bbe2-4110-8ac1-42e629c8be80\") " pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:06:01.176050 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.176033 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-home\") pod \"simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss\" (UID: \"49f28c58-bbe2-4110-8ac1-42e629c8be80\") " pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:06:01.176205 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.176190 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49f28c58-bbe2-4110-8ac1-42e629c8be80-tls-certs\") pod \"simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss\" (UID: \"49f28c58-bbe2-4110-8ac1-42e629c8be80\") " pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:06:01.176341 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.176324 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gv62\" (UniqueName: \"kubernetes.io/projected/49f28c58-bbe2-4110-8ac1-42e629c8be80-kube-api-access-7gv62\") pod \"simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss\" (UID: \"49f28c58-bbe2-4110-8ac1-42e629c8be80\") " pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:06:01.176572 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.176555 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-tmp-dir\") pod \"simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss\" (UID: \"49f28c58-bbe2-4110-8ac1-42e629c8be80\") " pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:06:01.176723 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.176708 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-model-cache\") pod \"simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss\" (UID: \"49f28c58-bbe2-4110-8ac1-42e629c8be80\") " pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:06:01.177266 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.177231 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-model-cache\") pod \"simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss\" (UID: \"49f28c58-bbe2-4110-8ac1-42e629c8be80\") " pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:06:01.177694 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.177675 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-tmp-dir\") pod \"simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss\" (UID: \"49f28c58-bbe2-4110-8ac1-42e629c8be80\") " pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:06:01.178122 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.178102 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-home\") pod \"simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss\" (UID: \"49f28c58-bbe2-4110-8ac1-42e629c8be80\") " pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:06:01.195123 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.192801 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-dshm\") pod \"simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss\" (UID: \"49f28c58-bbe2-4110-8ac1-42e629c8be80\") " pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:06:01.209340 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.209242 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49f28c58-bbe2-4110-8ac1-42e629c8be80-tls-certs\") pod \"simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss\" (UID: \"49f28c58-bbe2-4110-8ac1-42e629c8be80\") " pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:06:01.243292 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.236565 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gv62\" (UniqueName: \"kubernetes.io/projected/49f28c58-bbe2-4110-8ac1-42e629c8be80-kube-api-access-7gv62\") pod \"simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss\" (UID: \"49f28c58-bbe2-4110-8ac1-42e629c8be80\") " pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:06:01.442628 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.442583 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:06:01.828824 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:01.828797 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss"] Oct 30 22:06:01.830957 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:06:01.830921 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49f28c58_bbe2_4110_8ac1_42e629c8be80.slice/crio-eba0f8cb5747fa8d31cfa80d6fd3932c72490219d64359cbf5f1e768a7264509 WatchSource:0}: Error finding container eba0f8cb5747fa8d31cfa80d6fd3932c72490219d64359cbf5f1e768a7264509: Status 404 returned error can't find the container with id eba0f8cb5747fa8d31cfa80d6fd3932c72490219d64359cbf5f1e768a7264509 Oct 30 22:06:02.371346 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:02.371301 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" event={"ID":"49f28c58-bbe2-4110-8ac1-42e629c8be80","Type":"ContainerStarted","Data":"1c9a8e6cf55ecbd8b352c708dcb5dfb9ce34d089ea5963246c051ee537cba547"} Oct 30 22:06:02.371346 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:02.371351 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" event={"ID":"49f28c58-bbe2-4110-8ac1-42e629c8be80","Type":"ContainerStarted","Data":"eba0f8cb5747fa8d31cfa80d6fd3932c72490219d64359cbf5f1e768a7264509"} Oct 30 22:06:02.442931 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:02.442894 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:06:02.483656 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:02.483597 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podStartSLOduration=2.483581586 podStartE2EDuration="2.483581586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:02.482747636 +0000 UTC m=+965.034923951" watchObservedRunningTime="2026-10-30 22:06:02.483581586 +0000 UTC m=+965.035757885" Oct 30 22:06:06.941042 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:06.941004 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk"] Oct 30 22:06:06.945039 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:06.945019 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:06:07.039697 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:07.039657 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-home\") pod \"premium-w2-bc714d-kserve-755cdc9757-xqfvk\" (UID: \"a9fc3151-cd09-4454-89c0-dc57204054ca\") " pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:06:07.039878 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:07.039710 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-tmp-dir\") pod \"premium-w2-bc714d-kserve-755cdc9757-xqfvk\" (UID: \"a9fc3151-cd09-4454-89c0-dc57204054ca\") " pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:06:07.039878 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:07.039746 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4db9n\" (UniqueName: \"kubernetes.io/projected/a9fc3151-cd09-4454-89c0-dc57204054ca-kube-api-access-4db9n\") pod \"premium-w2-bc714d-kserve-755cdc9757-xqfvk\" (UID: \"a9fc3151-cd09-4454-89c0-dc57204054ca\") " pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:06:07.039969 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:07.039956 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-dshm\") pod \"premium-w2-bc714d-kserve-755cdc9757-xqfvk\" (UID: \"a9fc3151-cd09-4454-89c0-dc57204054ca\") " pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:06:07.040010 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:07.039993 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-model-cache\") pod \"premium-w2-bc714d-kserve-755cdc9757-xqfvk\" (UID: \"a9fc3151-cd09-4454-89c0-dc57204054ca\") " pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:06:07.040102 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:07.040080 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9fc3151-cd09-4454-89c0-dc57204054ca-tls-certs\") pod \"premium-w2-bc714d-kserve-755cdc9757-xqfvk\" (UID: \"a9fc3151-cd09-4454-89c0-dc57204054ca\") " pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:06:07.041183 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:07.041162 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-bc714d\"/\"premium-w2-bc714d-kserve-self-signed-certs\"" Oct 30 22:06:07.043116 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:07.043095 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk"] Oct 30 22:06:07.140859 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:07.140823 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9fc3151-cd09-4454-89c0-dc57204054ca-tls-certs\") pod \"premium-w2-bc714d-kserve-755cdc9757-xqfvk\" (UID: \"a9fc3151-cd09-4454-89c0-dc57204054ca\") " pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:06:07.141076 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:07.140923 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-home\") pod \"premium-w2-bc714d-kserve-755cdc9757-xqfvk\" (UID: \"a9fc3151-cd09-4454-89c0-dc57204054ca\") " pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:06:07.141076 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:07.140972 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-tmp-dir\") pod \"premium-w2-bc714d-kserve-755cdc9757-xqfvk\" (UID: \"a9fc3151-cd09-4454-89c0-dc57204054ca\") " pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:06:07.141076 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:07.141029 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4db9n\" (UniqueName: \"kubernetes.io/projected/a9fc3151-cd09-4454-89c0-dc57204054ca-kube-api-access-4db9n\") pod \"premium-w2-bc714d-kserve-755cdc9757-xqfvk\" (UID: \"a9fc3151-cd09-4454-89c0-dc57204054ca\") " pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:06:07.141076 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:07.141069 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-dshm\") pod \"premium-w2-bc714d-kserve-755cdc9757-xqfvk\" (UID: \"a9fc3151-cd09-4454-89c0-dc57204054ca\") " pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:06:07.141300 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:07.141111 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-model-cache\") pod \"premium-w2-bc714d-kserve-755cdc9757-xqfvk\" (UID: \"a9fc3151-cd09-4454-89c0-dc57204054ca\") " pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:06:07.141566 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:07.141543 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-home\") pod \"premium-w2-bc714d-kserve-755cdc9757-xqfvk\" (UID: \"a9fc3151-cd09-4454-89c0-dc57204054ca\") " pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:06:07.142216 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:07.142192 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-tmp-dir\") pod \"premium-w2-bc714d-kserve-755cdc9757-xqfvk\" (UID: \"a9fc3151-cd09-4454-89c0-dc57204054ca\") " pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:06:07.142368 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:07.142344 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-model-cache\") pod \"premium-w2-bc714d-kserve-755cdc9757-xqfvk\" (UID: \"a9fc3151-cd09-4454-89c0-dc57204054ca\") " pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:06:07.143920 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:07.143894 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9fc3151-cd09-4454-89c0-dc57204054ca-tls-certs\") pod \"premium-w2-bc714d-kserve-755cdc9757-xqfvk\" (UID: \"a9fc3151-cd09-4454-89c0-dc57204054ca\") " pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:06:07.144178 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:07.144160 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-dshm\") pod \"premium-w2-bc714d-kserve-755cdc9757-xqfvk\" (UID: \"a9fc3151-cd09-4454-89c0-dc57204054ca\") " pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:06:07.200048 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:07.199964 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4db9n\" (UniqueName: \"kubernetes.io/projected/a9fc3151-cd09-4454-89c0-dc57204054ca-kube-api-access-4db9n\") pod \"premium-w2-bc714d-kserve-755cdc9757-xqfvk\" (UID: \"a9fc3151-cd09-4454-89c0-dc57204054ca\") " pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:06:07.258020 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:07.257981 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:06:07.657861 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:07.657832 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk"] Oct 30 22:06:07.659875 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:06:07.659848 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9fc3151_cd09_4454_89c0_dc57204054ca.slice/crio-e59ac3e7705b80edfb1fe162afdae4be81250780345d62b364f1535de1d30d3a WatchSource:0}: Error finding container e59ac3e7705b80edfb1fe162afdae4be81250780345d62b364f1535de1d30d3a: Status 404 returned error can't find the container with id e59ac3e7705b80edfb1fe162afdae4be81250780345d62b364f1535de1d30d3a Oct 30 22:06:08.404814 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:08.404774 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" event={"ID":"a9fc3151-cd09-4454-89c0-dc57204054ca","Type":"ContainerStarted","Data":"98806554a9c31588ac1844508be03fb6345e9d1f18a47b8492d7f12e5d665478"} Oct 30 22:06:08.405227 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:08.404827 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" event={"ID":"a9fc3151-cd09-4454-89c0-dc57204054ca","Type":"ContainerStarted","Data":"e59ac3e7705b80edfb1fe162afdae4be81250780345d62b364f1535de1d30d3a"} Oct 30 22:06:08.518621 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:08.518562 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podStartSLOduration=2.5185428180000002 podStartE2EDuration="2.518542818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:08.517165624 +0000 UTC m=+971.069341917" watchObservedRunningTime="2026-10-30 22:06:08.518542818 +0000 UTC m=+971.070719119" Oct 30 22:06:09.258752 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:09.258712 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:06:11.443767 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:11.443726 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:06:11.461610 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:11.461583 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:06:12.445366 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:12.445333 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:06:12.799420 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:12.799374 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:06:17.259047 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:17.259010 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:06:17.272077 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:17.272043 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:06:17.467270 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:17.467224 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:06:27.910037 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:06:27.909996 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:06:35.752695 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:35.752606 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:06:54.117118 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:54.117078 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:06:55.202543 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:55.202507 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn"] Oct 30 22:06:55.206773 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:55.206747 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:06:55.303140 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:55.303088 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a00d9b\"/\"premium-w1-a00d9b-kserve-self-signed-certs\"" Oct 30 22:06:55.306459 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:55.306424 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn"] Oct 30 22:06:55.343843 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:55.343675 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-tmp-dir\") pod \"premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn\" (UID: \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\") " pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:06:55.343843 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:55.343784 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-tls-certs\") pod \"premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn\" (UID: \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\") " pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:06:55.343843 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:55.343850 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-dshm\") pod \"premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn\" (UID: \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\") " pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:06:55.344182 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:55.343874 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-home\") pod \"premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn\" (UID: \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\") " pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:06:55.344182 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:55.343899 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-model-cache\") pod \"premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn\" (UID: \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\") " pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:06:55.344182 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:55.344129 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nphht\" (UniqueName: \"kubernetes.io/projected/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-kube-api-access-nphht\") pod \"premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn\" (UID: \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\") " pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:06:55.445574 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:55.445535 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-tls-certs\") pod \"premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn\" (UID: \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\") " pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:06:55.445778 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:55.445594 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-dshm\") pod \"premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn\" (UID: \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\") " pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:06:55.445778 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:55.445611 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-home\") pod \"premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn\" (UID: \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\") " pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:06:55.445778 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:55.445628 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-model-cache\") pod \"premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn\" (UID: \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\") " pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:06:55.445778 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:55.445723 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nphht\" (UniqueName: \"kubernetes.io/projected/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-kube-api-access-nphht\") pod \"premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn\" (UID: \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\") " pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:06:55.445778 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:55.445778 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-tmp-dir\") pod \"premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn\" (UID: \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\") " pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:06:55.446134 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:55.446102 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-home\") pod \"premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn\" (UID: \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\") " pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:06:55.446275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:55.446115 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-tmp-dir\") pod \"premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn\" (UID: \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\") " pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:06:55.446554 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:55.446526 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-model-cache\") pod \"premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn\" (UID: \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\") " pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:06:55.448481 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:55.448458 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-dshm\") pod \"premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn\" (UID: \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\") " pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:06:55.448645 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:55.448477 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-tls-certs\") pod \"premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn\" (UID: \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\") " pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:06:55.504879 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:55.504789 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nphht\" (UniqueName: \"kubernetes.io/projected/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-kube-api-access-nphht\") pod \"premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn\" (UID: \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\") " pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:06:55.521466 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:55.521423 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:06:55.902912 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:55.902886 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn"] Oct 30 22:06:55.905107 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:06:55.905082 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb90b6498_0e99_4a90_b2ae_1bc4f2446d5b.slice/crio-33049ee133310a074ae59cc2a097032e096608bb3e3d4e3b968871bf168e241c WatchSource:0}: Error finding container 33049ee133310a074ae59cc2a097032e096608bb3e3d4e3b968871bf168e241c: Status 404 returned error can't find the container with id 33049ee133310a074ae59cc2a097032e096608bb3e3d4e3b968871bf168e241c Oct 30 22:06:56.693153 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:56.693112 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" event={"ID":"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b","Type":"ContainerStarted","Data":"c0d8465ee82e660a07575a51d1134dccdc1b9a394f57c5ee90613a704e83d89e"} Oct 30 22:06:56.693597 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:56.693174 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" event={"ID":"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b","Type":"ContainerStarted","Data":"33049ee133310a074ae59cc2a097032e096608bb3e3d4e3b968871bf168e241c"} Oct 30 22:06:56.805311 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:56.805238 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podStartSLOduration=1.8052238630000002 podStartE2EDuration="1.805223863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:56.804209245 +0000 UTC m=+1019.356385558" watchObservedRunningTime="2026-10-30 22:06:56.805223863 +0000 UTC m=+1019.357400164" Oct 30 22:06:57.521920 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:06:57.521878 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:06:57.910287 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:06:57.910231 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:07:05.522733 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:05.522692 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:07:05.535524 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:05.535495 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:07:05.763833 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:05.763797 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:07:18.430333 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:18.430288 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:07:27.911074 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:07:27.911035 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:07:36.790232 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:36.790192 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g"] Oct 30 22:07:36.794492 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:36.794469 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:07:36.890830 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:36.890795 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c9dbe6\"/\"premium-w5-c9dbe6-kserve-self-signed-certs\"" Oct 30 22:07:36.892730 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:36.892704 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g"] Oct 30 22:07:36.903168 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:36.903125 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-tmp-dir\") pod \"premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g\" (UID: \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\") " pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:07:36.903359 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:36.903179 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-model-cache\") pod \"premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g\" (UID: \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\") " pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:07:36.903405 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:36.903364 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-home\") pod \"premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g\" (UID: \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\") " pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:07:36.903405 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:36.903397 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lqgs\" (UniqueName: \"kubernetes.io/projected/21c2b5d2-ef4b-4b68-af47-d550a11385d7-kube-api-access-4lqgs\") pod \"premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g\" (UID: \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\") " pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:07:36.903489 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:36.903477 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21c2b5d2-ef4b-4b68-af47-d550a11385d7-tls-certs\") pod \"premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g\" (UID: \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\") " pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:07:36.903525 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:36.903510 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-dshm\") pod \"premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g\" (UID: \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\") " pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:07:37.004514 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:37.004457 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-tmp-dir\") pod \"premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g\" (UID: \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\") " pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:07:37.004514 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:37.004517 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-model-cache\") pod \"premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g\" (UID: \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\") " pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:07:37.004787 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:37.004607 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-home\") pod \"premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g\" (UID: \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\") " pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:07:37.004787 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:37.004634 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lqgs\" (UniqueName: \"kubernetes.io/projected/21c2b5d2-ef4b-4b68-af47-d550a11385d7-kube-api-access-4lqgs\") pod \"premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g\" (UID: \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\") " pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:07:37.004787 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:37.004702 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21c2b5d2-ef4b-4b68-af47-d550a11385d7-tls-certs\") pod \"premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g\" (UID: \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\") " pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:07:37.004787 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:37.004739 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-dshm\") pod \"premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g\" (UID: \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\") " pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:07:37.005084 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:37.004985 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-tmp-dir\") pod \"premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g\" (UID: \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\") " pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:07:37.005084 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:37.004999 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-model-cache\") pod \"premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g\" (UID: \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\") " pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:07:37.005299 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:37.005268 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-home\") pod \"premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g\" (UID: \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\") " pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:07:37.007777 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:37.007745 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-dshm\") pod \"premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g\" (UID: \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\") " pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:07:37.008029 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:37.008008 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21c2b5d2-ef4b-4b68-af47-d550a11385d7-tls-certs\") pod \"premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g\" (UID: \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\") " pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:07:37.064435 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:37.064358 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lqgs\" (UniqueName: \"kubernetes.io/projected/21c2b5d2-ef4b-4b68-af47-d550a11385d7-kube-api-access-4lqgs\") pod \"premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g\" (UID: \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\") " pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:07:37.106786 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:37.106744 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:07:37.511691 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:37.511664 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g"] Oct 30 22:07:37.514645 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:07:37.514614 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21c2b5d2_ef4b_4b68_af47_d550a11385d7.slice/crio-6388b1a5a04868dc1e0cf6cd1a32edf7f5cd106e0b811d59d5209b83a78ced3b WatchSource:0}: Error finding container 6388b1a5a04868dc1e0cf6cd1a32edf7f5cd106e0b811d59d5209b83a78ced3b: Status 404 returned error can't find the container with id 6388b1a5a04868dc1e0cf6cd1a32edf7f5cd106e0b811d59d5209b83a78ced3b Oct 30 22:07:37.950643 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:37.950600 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" event={"ID":"21c2b5d2-ef4b-4b68-af47-d550a11385d7","Type":"ContainerStarted","Data":"b271c93432210844c022fbe220ab3c64946294a1fbb5338ae2eb145ec5b07e9f"} Oct 30 22:07:37.951102 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:37.950654 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" event={"ID":"21c2b5d2-ef4b-4b68-af47-d550a11385d7","Type":"ContainerStarted","Data":"6388b1a5a04868dc1e0cf6cd1a32edf7f5cd106e0b811d59d5209b83a78ced3b"} Oct 30 22:07:38.064811 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:38.064740 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podStartSLOduration=2.064720039 podStartE2EDuration="2.064720039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:38.0633048 +0000 UTC m=+1060.615481078" watchObservedRunningTime="2026-10-30 22:07:38.064720039 +0000 UTC m=+1060.616896361" Oct 30 22:07:38.107636 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:38.107583 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:07:47.107709 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:47.107672 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:07:47.120543 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:47.120507 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:07:48.035662 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:48.035625 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:07:51.206122 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:51.206083 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:07:51.842599 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:51.842550 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2c023d8-mg7lg"] Oct 30 22:07:51.847449 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:51.847419 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c023d8-mg7lg" Oct 30 22:07:51.899372 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:51.899334 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:07:51.902847 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:51.902806 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2c023d8-mg7lg"] Oct 30 22:07:51.946746 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:51.946710 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:07:51.979751 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:51.979710 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d09cbbc8-a913-47a1-aee5-c9ee6771abe8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2c023d8-mg7lg\" (UID: \"d09cbbc8-a913-47a1-aee5-c9ee6771abe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c023d8-mg7lg" Oct 30 22:07:51.979941 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:51.979893 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtvr4\" (UniqueName: \"kubernetes.io/projected/d09cbbc8-a913-47a1-aee5-c9ee6771abe8-kube-api-access-qtvr4\") pod \"maas-api-revoke-sub-c2c023d8-mg7lg\" (UID: \"d09cbbc8-a913-47a1-aee5-c9ee6771abe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c023d8-mg7lg" Oct 30 22:07:51.979993 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:51.979951 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d09cbbc8-a913-47a1-aee5-c9ee6771abe8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2c023d8-mg7lg\" (UID: \"d09cbbc8-a913-47a1-aee5-c9ee6771abe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c023d8-mg7lg" Oct 30 22:07:52.080787 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:52.080668 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d09cbbc8-a913-47a1-aee5-c9ee6771abe8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2c023d8-mg7lg\" (UID: \"d09cbbc8-a913-47a1-aee5-c9ee6771abe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c023d8-mg7lg" Oct 30 22:07:52.080998 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:52.080867 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtvr4\" (UniqueName: \"kubernetes.io/projected/d09cbbc8-a913-47a1-aee5-c9ee6771abe8-kube-api-access-qtvr4\") pod \"maas-api-revoke-sub-c2c023d8-mg7lg\" (UID: \"d09cbbc8-a913-47a1-aee5-c9ee6771abe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c023d8-mg7lg" Oct 30 22:07:52.080998 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:52.080921 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d09cbbc8-a913-47a1-aee5-c9ee6771abe8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2c023d8-mg7lg\" (UID: \"d09cbbc8-a913-47a1-aee5-c9ee6771abe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c023d8-mg7lg" Oct 30 22:07:52.081693 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:52.081665 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d09cbbc8-a913-47a1-aee5-c9ee6771abe8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2c023d8-mg7lg\" (UID: \"d09cbbc8-a913-47a1-aee5-c9ee6771abe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c023d8-mg7lg" Oct 30 22:07:52.083601 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:52.083570 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d09cbbc8-a913-47a1-aee5-c9ee6771abe8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2c023d8-mg7lg\" (UID: \"d09cbbc8-a913-47a1-aee5-c9ee6771abe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c023d8-mg7lg" Oct 30 22:07:52.140475 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:52.140396 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtvr4\" (UniqueName: \"kubernetes.io/projected/d09cbbc8-a913-47a1-aee5-c9ee6771abe8-kube-api-access-qtvr4\") pod \"maas-api-revoke-sub-c2c023d8-mg7lg\" (UID: \"d09cbbc8-a913-47a1-aee5-c9ee6771abe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c023d8-mg7lg" Oct 30 22:07:52.162667 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:52.162627 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c023d8-mg7lg" Oct 30 22:07:52.539084 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:52.539055 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2c023d8-mg7lg"] Oct 30 22:07:52.541058 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:07:52.541026 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd09cbbc8_a913_47a1_aee5_c9ee6771abe8.slice/crio-921461cb23ae1dec6a99cee9c72dda76873b43496c38107bfe90537912eb0aa9 WatchSource:0}: Error finding container 921461cb23ae1dec6a99cee9c72dda76873b43496c38107bfe90537912eb0aa9: Status 404 returned error can't find the container with id 921461cb23ae1dec6a99cee9c72dda76873b43496c38107bfe90537912eb0aa9 Oct 30 22:07:53.053244 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:53.053204 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c023d8-mg7lg" event={"ID":"d09cbbc8-a913-47a1-aee5-c9ee6771abe8","Type":"ContainerStarted","Data":"8fc1bf6808144cc21645e271600cc70f1982eb3785cb142e298bc0fd8f0fb5a7"} Oct 30 22:07:53.053474 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:53.053291 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c023d8-mg7lg" event={"ID":"d09cbbc8-a913-47a1-aee5-c9ee6771abe8","Type":"ContainerStarted","Data":"921461cb23ae1dec6a99cee9c72dda76873b43496c38107bfe90537912eb0aa9"} Oct 30 22:07:53.165234 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:53.165120 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c023d8-mg7lg" podStartSLOduration=2.165105659 podStartE2EDuration="2.165105659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:53.163687629 +0000 UTC m=+1075.715863929" watchObservedRunningTime="2026-10-30 22:07:53.165105659 +0000 UTC m=+1075.717281959" Oct 30 22:07:54.059940 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:54.059898 6006 generic.go:356] "Generic (PLEG): container finished" podID="d09cbbc8-a913-47a1-aee5-c9ee6771abe8" containerID="8fc1bf6808144cc21645e271600cc70f1982eb3785cb142e298bc0fd8f0fb5a7" exitCode=0 Oct 30 22:07:54.060369 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:54.059979 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c023d8-mg7lg" event={"ID":"d09cbbc8-a913-47a1-aee5-c9ee6771abe8","Type":"ContainerDied","Data":"8fc1bf6808144cc21645e271600cc70f1982eb3785cb142e298bc0fd8f0fb5a7"} Oct 30 22:07:55.207616 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:55.207583 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c023d8-mg7lg" Oct 30 22:07:55.316322 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:55.316211 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d09cbbc8-a913-47a1-aee5-c9ee6771abe8-kube-api-access-qtvr4\" (UniqueName: \"kubernetes.io/projected/d09cbbc8-a913-47a1-aee5-c9ee6771abe8-kube-api-access-qtvr4\") pod \"d09cbbc8-a913-47a1-aee5-c9ee6771abe8\" (UID: \"d09cbbc8-a913-47a1-aee5-c9ee6771abe8\") " Oct 30 22:07:55.316488 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:55.316355 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d09cbbc8-a913-47a1-aee5-c9ee6771abe8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d09cbbc8-a913-47a1-aee5-c9ee6771abe8-maas-api-openshift-ca\") pod \"d09cbbc8-a913-47a1-aee5-c9ee6771abe8\" (UID: \"d09cbbc8-a913-47a1-aee5-c9ee6771abe8\") " Oct 30 22:07:55.316488 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:55.316418 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d09cbbc8-a913-47a1-aee5-c9ee6771abe8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d09cbbc8-a913-47a1-aee5-c9ee6771abe8-maas-api-xks-ca\") pod \"d09cbbc8-a913-47a1-aee5-c9ee6771abe8\" (UID: \"d09cbbc8-a913-47a1-aee5-c9ee6771abe8\") " Oct 30 22:07:55.316726 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:55.316690 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d09cbbc8-a913-47a1-aee5-c9ee6771abe8-maas-api-openshift-ca" pod "d09cbbc8-a913-47a1-aee5-c9ee6771abe8" (UID: "d09cbbc8-a913-47a1-aee5-c9ee6771abe8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:55.319006 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:55.318974 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d09cbbc8-a913-47a1-aee5-c9ee6771abe8-maas-api-xks-ca" pod "d09cbbc8-a913-47a1-aee5-c9ee6771abe8" (UID: "d09cbbc8-a913-47a1-aee5-c9ee6771abe8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:55.319696 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:55.319665 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d09cbbc8-a913-47a1-aee5-c9ee6771abe8-kube-api-access-qtvr4" pod "d09cbbc8-a913-47a1-aee5-c9ee6771abe8" (UID: "d09cbbc8-a913-47a1-aee5-c9ee6771abe8"). InnerVolumeSpecName "kube-api-access-qtvr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:55.417908 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:55.417864 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtvr4\" (UniqueName: \"kubernetes.io/projected/d09cbbc8-a913-47a1-aee5-c9ee6771abe8-kube-api-access-qtvr4\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:55.417908 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:55.417902 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d09cbbc8-a913-47a1-aee5-c9ee6771abe8-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:55.417908 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:55.417913 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d09cbbc8-a913-47a1-aee5-c9ee6771abe8-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:56.073324 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:56.073285 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c023d8-mg7lg" event={"ID":"d09cbbc8-a913-47a1-aee5-c9ee6771abe8","Type":"ContainerDied","Data":"921461cb23ae1dec6a99cee9c72dda76873b43496c38107bfe90537912eb0aa9"} Oct 30 22:07:56.073512 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:56.073346 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="921461cb23ae1dec6a99cee9c72dda76873b43496c38107bfe90537912eb0aa9" Oct 30 22:07:56.073512 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:07:56.073306 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c023d8-mg7lg" Oct 30 22:07:57.910997 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:07:57.910949 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:08:07.953655 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:07.953569 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b293ccec-62fvd"] Oct 30 22:08:07.954165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:07.954058 6006 cpu_manager.go:414] "containerMap: removing container" podUID="d09cbbc8-a913-47a1-aee5-c9ee6771abe8" containerName="revoke-keys" Oct 30 22:08:07.954165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:07.954070 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d09cbbc8-a913-47a1-aee5-c9ee6771abe8" containerName="revoke-keys" Oct 30 22:08:07.954165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:07.954162 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="d09cbbc8-a913-47a1-aee5-c9ee6771abe8" containerName="revoke-keys" Oct 30 22:08:07.958564 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:07.958543 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b293ccec-62fvd" Oct 30 22:08:08.015378 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:08.014017 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b293ccec-62fvd"] Oct 30 22:08:08.057745 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:08.057705 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:08:08.057913 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:08.057825 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:08:08.062054 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:08.062020 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86404af4-977e-4617-ab0d-18a350387d01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b293ccec-62fvd\" (UID: \"86404af4-977e-4617-ab0d-18a350387d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b293ccec-62fvd" Oct 30 22:08:08.062231 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:08.062152 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86404af4-977e-4617-ab0d-18a350387d01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b293ccec-62fvd\" (UID: \"86404af4-977e-4617-ab0d-18a350387d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b293ccec-62fvd" Oct 30 22:08:08.062231 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:08.062196 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g9pp\" (UniqueName: \"kubernetes.io/projected/86404af4-977e-4617-ab0d-18a350387d01-kube-api-access-6g9pp\") pod \"maas-api-revoke-sub-b293ccec-62fvd\" (UID: \"86404af4-977e-4617-ab0d-18a350387d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b293ccec-62fvd" Oct 30 22:08:08.163663 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:08.163626 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86404af4-977e-4617-ab0d-18a350387d01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b293ccec-62fvd\" (UID: \"86404af4-977e-4617-ab0d-18a350387d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b293ccec-62fvd" Oct 30 22:08:08.163852 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:08.163673 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g9pp\" (UniqueName: \"kubernetes.io/projected/86404af4-977e-4617-ab0d-18a350387d01-kube-api-access-6g9pp\") pod \"maas-api-revoke-sub-b293ccec-62fvd\" (UID: \"86404af4-977e-4617-ab0d-18a350387d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b293ccec-62fvd" Oct 30 22:08:08.163852 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:08.163749 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86404af4-977e-4617-ab0d-18a350387d01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b293ccec-62fvd\" (UID: \"86404af4-977e-4617-ab0d-18a350387d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b293ccec-62fvd" Oct 30 22:08:08.164464 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:08.164440 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86404af4-977e-4617-ab0d-18a350387d01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b293ccec-62fvd\" (UID: \"86404af4-977e-4617-ab0d-18a350387d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b293ccec-62fvd" Oct 30 22:08:08.166108 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:08.166079 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86404af4-977e-4617-ab0d-18a350387d01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b293ccec-62fvd\" (UID: \"86404af4-977e-4617-ab0d-18a350387d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b293ccec-62fvd" Oct 30 22:08:08.222688 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:08.222612 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g9pp\" (UniqueName: \"kubernetes.io/projected/86404af4-977e-4617-ab0d-18a350387d01-kube-api-access-6g9pp\") pod \"maas-api-revoke-sub-b293ccec-62fvd\" (UID: \"86404af4-977e-4617-ab0d-18a350387d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b293ccec-62fvd" Oct 30 22:08:08.272351 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:08.272318 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b293ccec-62fvd" Oct 30 22:08:08.647133 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:08.647109 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b293ccec-62fvd"] Oct 30 22:08:08.649696 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:08:08.649667 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86404af4_977e_4617_ab0d_18a350387d01.slice/crio-b417c92a12095cce57b84bfa93beeaedb107cc68102ceef4a76e3d84ace7772c WatchSource:0}: Error finding container b417c92a12095cce57b84bfa93beeaedb107cc68102ceef4a76e3d84ace7772c: Status 404 returned error can't find the container with id b417c92a12095cce57b84bfa93beeaedb107cc68102ceef4a76e3d84ace7772c Oct 30 22:08:09.161299 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:09.161200 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b293ccec-62fvd" event={"ID":"86404af4-977e-4617-ab0d-18a350387d01","Type":"ContainerStarted","Data":"871bf540b36b8ac79b11b15019f81dad6c29adc3021249f6e077ca5fb516909a"} Oct 30 22:08:09.161299 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:09.161265 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b293ccec-62fvd" event={"ID":"86404af4-977e-4617-ab0d-18a350387d01","Type":"ContainerStarted","Data":"b417c92a12095cce57b84bfa93beeaedb107cc68102ceef4a76e3d84ace7772c"} Oct 30 22:08:09.272749 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:09.272679 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b293ccec-62fvd" podStartSLOduration=2.272659873 podStartE2EDuration="2.272659873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:09.271559148 +0000 UTC m=+1091.823735450" watchObservedRunningTime="2026-10-30 22:08:09.272659873 +0000 UTC m=+1091.824836176" Oct 30 22:08:10.168140 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:10.168091 6006 generic.go:356] "Generic (PLEG): container finished" podID="86404af4-977e-4617-ab0d-18a350387d01" containerID="871bf540b36b8ac79b11b15019f81dad6c29adc3021249f6e077ca5fb516909a" exitCode=0 Oct 30 22:08:10.168574 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:10.168185 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b293ccec-62fvd" event={"ID":"86404af4-977e-4617-ab0d-18a350387d01","Type":"ContainerDied","Data":"871bf540b36b8ac79b11b15019f81dad6c29adc3021249f6e077ca5fb516909a"} Oct 30 22:08:11.346087 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:11.346058 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b293ccec-62fvd" Oct 30 22:08:11.500299 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:11.500185 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86404af4-977e-4617-ab0d-18a350387d01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86404af4-977e-4617-ab0d-18a350387d01-maas-api-xks-ca\") pod \"86404af4-977e-4617-ab0d-18a350387d01\" (UID: \"86404af4-977e-4617-ab0d-18a350387d01\") " Oct 30 22:08:11.500299 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:11.500278 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86404af4-977e-4617-ab0d-18a350387d01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86404af4-977e-4617-ab0d-18a350387d01-maas-api-openshift-ca\") pod \"86404af4-977e-4617-ab0d-18a350387d01\" (UID: \"86404af4-977e-4617-ab0d-18a350387d01\") " Oct 30 22:08:11.500534 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:11.500321 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86404af4-977e-4617-ab0d-18a350387d01-kube-api-access-6g9pp\" (UniqueName: \"kubernetes.io/projected/86404af4-977e-4617-ab0d-18a350387d01-kube-api-access-6g9pp\") pod \"86404af4-977e-4617-ab0d-18a350387d01\" (UID: \"86404af4-977e-4617-ab0d-18a350387d01\") " Oct 30 22:08:11.500992 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:11.500944 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86404af4-977e-4617-ab0d-18a350387d01-maas-api-openshift-ca" pod "86404af4-977e-4617-ab0d-18a350387d01" (UID: "86404af4-977e-4617-ab0d-18a350387d01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:11.503045 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:11.503014 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86404af4-977e-4617-ab0d-18a350387d01-kube-api-access-6g9pp" pod "86404af4-977e-4617-ab0d-18a350387d01" (UID: "86404af4-977e-4617-ab0d-18a350387d01"). InnerVolumeSpecName "kube-api-access-6g9pp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:11.503473 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:11.503444 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86404af4-977e-4617-ab0d-18a350387d01-maas-api-xks-ca" pod "86404af4-977e-4617-ab0d-18a350387d01" (UID: "86404af4-977e-4617-ab0d-18a350387d01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:11.601523 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:11.601484 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86404af4-977e-4617-ab0d-18a350387d01-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:11.601523 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:11.601522 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86404af4-977e-4617-ab0d-18a350387d01-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:11.601800 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:11.601537 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g9pp\" (UniqueName: \"kubernetes.io/projected/86404af4-977e-4617-ab0d-18a350387d01-kube-api-access-6g9pp\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:12.185654 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:12.185447 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b293ccec-62fvd" event={"ID":"86404af4-977e-4617-ab0d-18a350387d01","Type":"ContainerDied","Data":"b417c92a12095cce57b84bfa93beeaedb107cc68102ceef4a76e3d84ace7772c"} Oct 30 22:08:12.185654 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:12.185547 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b417c92a12095cce57b84bfa93beeaedb107cc68102ceef4a76e3d84ace7772c" Oct 30 22:08:12.185654 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:12.185618 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b293ccec-62fvd" Oct 30 22:08:12.343613 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:12.343573 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:08:13.932524 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:13.932480 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5274f32b-2ph2b"] Oct 30 22:08:13.935792 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:13.935762 6006 cpu_manager.go:414] "containerMap: removing container" podUID="86404af4-977e-4617-ab0d-18a350387d01" containerName="revoke-keys" Oct 30 22:08:13.935792 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:13.935791 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86404af4-977e-4617-ab0d-18a350387d01" containerName="revoke-keys" Oct 30 22:08:13.935959 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:13.935885 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="86404af4-977e-4617-ab0d-18a350387d01" containerName="revoke-keys" Oct 30 22:08:13.939504 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:13.939483 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5274f32b-2ph2b" Oct 30 22:08:13.991916 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:13.991876 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5274f32b-2ph2b"] Oct 30 22:08:14.035556 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:14.035499 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:08:14.035556 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:14.035499 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:08:14.129725 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:14.129673 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a8cc94b-62fb-4406-b0b4-bc383a2c892f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5274f32b-2ph2b\" (UID: \"6a8cc94b-62fb-4406-b0b4-bc383a2c892f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5274f32b-2ph2b" Oct 30 22:08:14.129922 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:14.129742 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a8cc94b-62fb-4406-b0b4-bc383a2c892f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5274f32b-2ph2b\" (UID: \"6a8cc94b-62fb-4406-b0b4-bc383a2c892f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5274f32b-2ph2b" Oct 30 22:08:14.129922 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:14.129856 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlkc2\" (UniqueName: \"kubernetes.io/projected/6a8cc94b-62fb-4406-b0b4-bc383a2c892f-kube-api-access-qlkc2\") pod \"maas-api-revoke-sub-5274f32b-2ph2b\" (UID: \"6a8cc94b-62fb-4406-b0b4-bc383a2c892f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5274f32b-2ph2b" Oct 30 22:08:14.231533 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:14.231425 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a8cc94b-62fb-4406-b0b4-bc383a2c892f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5274f32b-2ph2b\" (UID: \"6a8cc94b-62fb-4406-b0b4-bc383a2c892f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5274f32b-2ph2b" Oct 30 22:08:14.231533 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:14.231480 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a8cc94b-62fb-4406-b0b4-bc383a2c892f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5274f32b-2ph2b\" (UID: \"6a8cc94b-62fb-4406-b0b4-bc383a2c892f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5274f32b-2ph2b" Oct 30 22:08:14.231745 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:14.231536 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlkc2\" (UniqueName: \"kubernetes.io/projected/6a8cc94b-62fb-4406-b0b4-bc383a2c892f-kube-api-access-qlkc2\") pod \"maas-api-revoke-sub-5274f32b-2ph2b\" (UID: \"6a8cc94b-62fb-4406-b0b4-bc383a2c892f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5274f32b-2ph2b" Oct 30 22:08:14.232141 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:14.232113 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a8cc94b-62fb-4406-b0b4-bc383a2c892f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5274f32b-2ph2b\" (UID: \"6a8cc94b-62fb-4406-b0b4-bc383a2c892f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5274f32b-2ph2b" Oct 30 22:08:14.234106 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:14.234078 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a8cc94b-62fb-4406-b0b4-bc383a2c892f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5274f32b-2ph2b\" (UID: \"6a8cc94b-62fb-4406-b0b4-bc383a2c892f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5274f32b-2ph2b" Oct 30 22:08:14.290676 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:14.290646 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlkc2\" (UniqueName: \"kubernetes.io/projected/6a8cc94b-62fb-4406-b0b4-bc383a2c892f-kube-api-access-qlkc2\") pod \"maas-api-revoke-sub-5274f32b-2ph2b\" (UID: \"6a8cc94b-62fb-4406-b0b4-bc383a2c892f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5274f32b-2ph2b" Oct 30 22:08:14.552347 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:14.552307 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5274f32b-2ph2b" Oct 30 22:08:14.927544 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:14.927514 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5274f32b-2ph2b"] Oct 30 22:08:14.930676 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:08:14.930643 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a8cc94b_62fb_4406_b0b4_bc383a2c892f.slice/crio-df9f9d9d9d46e39f3dff7b69d437db36621de887b05d2d39107183d8f9f39405 WatchSource:0}: Error finding container df9f9d9d9d46e39f3dff7b69d437db36621de887b05d2d39107183d8f9f39405: Status 404 returned error can't find the container with id df9f9d9d9d46e39f3dff7b69d437db36621de887b05d2d39107183d8f9f39405 Oct 30 22:08:15.204750 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.204706 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5274f32b-2ph2b" event={"ID":"6a8cc94b-62fb-4406-b0b4-bc383a2c892f","Type":"ContainerStarted","Data":"df9f9d9d9d46e39f3dff7b69d437db36621de887b05d2d39107183d8f9f39405"} Oct 30 22:08:15.258753 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.258713 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4"] Oct 30 22:08:15.264070 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.264044 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.313953 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.313884 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-bc714d-openshift-default-dockercfg-gs4pf\"" Oct 30 22:08:15.347851 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.347803 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr28t\" (UniqueName: \"kubernetes.io/projected/efd2de68-03c7-4dc4-8422-a2a7f87aec11-kube-api-access-kr28t\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.348045 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.347863 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istiod-ca-cert\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.348045 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.347975 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-workload-certs\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.348045 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.348017 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-envoy\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.348229 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.348065 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-podinfo\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.348229 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.348126 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-credential-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.348229 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.348205 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-workload-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.348412 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.348237 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-token\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.348412 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.348307 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-data\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.363014 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.362978 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4"] Oct 30 22:08:15.449395 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.449352 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-workload-certs\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.449589 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.449412 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-envoy\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.449589 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.449447 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-podinfo\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.449589 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.449545 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-credential-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.449769 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.449606 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-workload-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.449769 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.449647 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-token\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.449769 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.449689 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-data\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.449769 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.449721 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr28t\" (UniqueName: \"kubernetes.io/projected/efd2de68-03c7-4dc4-8422-a2a7f87aec11-kube-api-access-kr28t\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.450038 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.449797 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istiod-ca-cert\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.450038 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.449827 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-workload-certs\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.450038 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.449936 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-credential-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.450038 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.450006 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-workload-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.450231 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.450128 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-data\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.450603 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.450581 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istiod-ca-cert\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.451898 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.451870 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-envoy\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.452215 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.452195 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-podinfo\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.510940 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.510830 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-token\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.512994 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.512959 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr28t\" (UniqueName: \"kubernetes.io/projected/efd2de68-03c7-4dc4-8422-a2a7f87aec11-kube-api-access-kr28t\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:15.578460 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:15.578418 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:16.022102 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:16.022067 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4"] Oct 30 22:08:16.023783 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:08:16.023756 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd2de68_03c7_4dc4_8422_a2a7f87aec11.slice/crio-6646ccf3ca673ac84d9f507533ad518a696c3ad1c8f3c6e5c452075e66b79597 WatchSource:0}: Error finding container 6646ccf3ca673ac84d9f507533ad518a696c3ad1c8f3c6e5c452075e66b79597: Status 404 returned error can't find the container with id 6646ccf3ca673ac84d9f507533ad518a696c3ad1c8f3c6e5c452075e66b79597 Oct 30 22:08:16.026936 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:16.026894 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:08:16.027006 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:16.026983 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:08:16.027046 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:16.027024 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:08:16.211948 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:16.211843 6006 generic.go:356] "Generic (PLEG): container finished" podID="6a8cc94b-62fb-4406-b0b4-bc383a2c892f" containerID="4c8aebbda243f8f92c82dfc893baacfe31f430d2d3d21fed33e36c6d0848e6ca" exitCode=0 Oct 30 22:08:16.211948 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:16.211909 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5274f32b-2ph2b" event={"ID":"6a8cc94b-62fb-4406-b0b4-bc383a2c892f","Type":"ContainerDied","Data":"4c8aebbda243f8f92c82dfc893baacfe31f430d2d3d21fed33e36c6d0848e6ca"} Oct 30 22:08:16.214101 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:16.214076 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" event={"ID":"efd2de68-03c7-4dc4-8422-a2a7f87aec11","Type":"ContainerStarted","Data":"88113bb66b9f3aa027181ffce88deb971c12b80208de7cba9fcba6c64685eb55"} Oct 30 22:08:16.214214 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:16.214119 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" event={"ID":"efd2de68-03c7-4dc4-8422-a2a7f87aec11","Type":"ContainerStarted","Data":"6646ccf3ca673ac84d9f507533ad518a696c3ad1c8f3c6e5c452075e66b79597"} Oct 30 22:08:16.446438 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:16.446386 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" podStartSLOduration=1.44637149 podStartE2EDuration="1.44637149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:16.44452815 +0000 UTC m=+1098.996704451" watchObservedRunningTime="2026-10-30 22:08:16.44637149 +0000 UTC m=+1098.998547790" Oct 30 22:08:16.579136 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:16.579091 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:17.392624 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:17.392596 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5274f32b-2ph2b" Oct 30 22:08:17.476544 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:17.476510 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a8cc94b-62fb-4406-b0b4-bc383a2c892f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a8cc94b-62fb-4406-b0b4-bc383a2c892f-maas-api-xks-ca\") pod \"6a8cc94b-62fb-4406-b0b4-bc383a2c892f\" (UID: \"6a8cc94b-62fb-4406-b0b4-bc383a2c892f\") " Oct 30 22:08:17.476720 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:17.476567 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a8cc94b-62fb-4406-b0b4-bc383a2c892f-kube-api-access-qlkc2\" (UniqueName: \"kubernetes.io/projected/6a8cc94b-62fb-4406-b0b4-bc383a2c892f-kube-api-access-qlkc2\") pod \"6a8cc94b-62fb-4406-b0b4-bc383a2c892f\" (UID: \"6a8cc94b-62fb-4406-b0b4-bc383a2c892f\") " Oct 30 22:08:17.476720 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:17.476603 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a8cc94b-62fb-4406-b0b4-bc383a2c892f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a8cc94b-62fb-4406-b0b4-bc383a2c892f-maas-api-openshift-ca\") pod \"6a8cc94b-62fb-4406-b0b4-bc383a2c892f\" (UID: \"6a8cc94b-62fb-4406-b0b4-bc383a2c892f\") " Oct 30 22:08:17.477055 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:17.477028 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a8cc94b-62fb-4406-b0b4-bc383a2c892f-maas-api-openshift-ca" pod "6a8cc94b-62fb-4406-b0b4-bc383a2c892f" (UID: "6a8cc94b-62fb-4406-b0b4-bc383a2c892f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:17.478669 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:17.478642 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a8cc94b-62fb-4406-b0b4-bc383a2c892f-maas-api-xks-ca" pod "6a8cc94b-62fb-4406-b0b4-bc383a2c892f" (UID: "6a8cc94b-62fb-4406-b0b4-bc383a2c892f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:17.478770 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:17.478645 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a8cc94b-62fb-4406-b0b4-bc383a2c892f-kube-api-access-qlkc2" pod "6a8cc94b-62fb-4406-b0b4-bc383a2c892f" (UID: "6a8cc94b-62fb-4406-b0b4-bc383a2c892f"). InnerVolumeSpecName "kube-api-access-qlkc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:17.577853 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:17.577812 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a8cc94b-62fb-4406-b0b4-bc383a2c892f-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:17.577853 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:17.577854 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlkc2\" (UniqueName: \"kubernetes.io/projected/6a8cc94b-62fb-4406-b0b4-bc383a2c892f-kube-api-access-qlkc2\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:17.578043 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:17.577871 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a8cc94b-62fb-4406-b0b4-bc383a2c892f-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:17.581101 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:17.581068 6006 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:08:17.581228 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:17.581143 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" podUID="efd2de68-03c7-4dc4-8422-a2a7f87aec11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 22:08:18.239406 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:18.239365 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5274f32b-2ph2b" event={"ID":"6a8cc94b-62fb-4406-b0b4-bc383a2c892f","Type":"ContainerDied","Data":"df9f9d9d9d46e39f3dff7b69d437db36621de887b05d2d39107183d8f9f39405"} Oct 30 22:08:18.239406 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:18.239390 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5274f32b-2ph2b" Oct 30 22:08:18.239623 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:18.239430 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df9f9d9d9d46e39f3dff7b69d437db36621de887b05d2d39107183d8f9f39405" Oct 30 22:08:18.579483 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:18.579442 6006 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:08:18.579892 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:18.579503 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" podUID="efd2de68-03c7-4dc4-8422-a2a7f87aec11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 22:08:19.579071 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:19.579029 6006 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:08:19.579301 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:19.579099 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" podUID="efd2de68-03c7-4dc4-8422-a2a7f87aec11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 22:08:20.579003 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:20.578960 6006 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:08:20.579437 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:20.579021 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" podUID="efd2de68-03c7-4dc4-8422-a2a7f87aec11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 22:08:21.579535 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:21.579495 6006 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:08:21.580084 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:21.579563 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" podUID="efd2de68-03c7-4dc4-8422-a2a7f87aec11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 22:08:22.738391 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:22.738328 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:22.738963 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:22.738934 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:22.739914 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:22.739890 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:08:24.767724 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:24.767690 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcca0367-cdxzf"] Oct 30 22:08:24.768186 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:24.768171 6006 cpu_manager.go:414] "containerMap: removing container" podUID="6a8cc94b-62fb-4406-b0b4-bc383a2c892f" containerName="revoke-keys" Oct 30 22:08:24.768280 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:24.768187 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a8cc94b-62fb-4406-b0b4-bc383a2c892f" containerName="revoke-keys" Oct 30 22:08:24.768324 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:24.768317 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a8cc94b-62fb-4406-b0b4-bc383a2c892f" containerName="revoke-keys" Oct 30 22:08:24.778951 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:24.778919 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcca0367-cdxzf" Oct 30 22:08:24.827383 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:24.827346 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcca0367-cdxzf"] Oct 30 22:08:24.879132 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:24.879089 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc00cea-a69e-4e48-90ef-c7a147696db8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcca0367-cdxzf\" (UID: \"1bc00cea-a69e-4e48-90ef-c7a147696db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcca0367-cdxzf" Oct 30 22:08:24.879335 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:24.879150 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q27hx\" (UniqueName: \"kubernetes.io/projected/1bc00cea-a69e-4e48-90ef-c7a147696db8-kube-api-access-q27hx\") pod \"maas-api-revoke-sub-bcca0367-cdxzf\" (UID: \"1bc00cea-a69e-4e48-90ef-c7a147696db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcca0367-cdxzf" Oct 30 22:08:24.879420 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:24.879390 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc00cea-a69e-4e48-90ef-c7a147696db8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcca0367-cdxzf\" (UID: \"1bc00cea-a69e-4e48-90ef-c7a147696db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcca0367-cdxzf" Oct 30 22:08:24.897068 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:24.897029 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:08:24.933334 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:24.933293 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:08:24.981018 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:24.980974 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc00cea-a69e-4e48-90ef-c7a147696db8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcca0367-cdxzf\" (UID: \"1bc00cea-a69e-4e48-90ef-c7a147696db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcca0367-cdxzf" Oct 30 22:08:24.981219 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:24.981104 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc00cea-a69e-4e48-90ef-c7a147696db8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcca0367-cdxzf\" (UID: \"1bc00cea-a69e-4e48-90ef-c7a147696db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcca0367-cdxzf" Oct 30 22:08:24.981219 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:24.981144 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q27hx\" (UniqueName: \"kubernetes.io/projected/1bc00cea-a69e-4e48-90ef-c7a147696db8-kube-api-access-q27hx\") pod \"maas-api-revoke-sub-bcca0367-cdxzf\" (UID: \"1bc00cea-a69e-4e48-90ef-c7a147696db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcca0367-cdxzf" Oct 30 22:08:24.981735 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:24.981715 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc00cea-a69e-4e48-90ef-c7a147696db8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcca0367-cdxzf\" (UID: \"1bc00cea-a69e-4e48-90ef-c7a147696db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcca0367-cdxzf" Oct 30 22:08:24.983614 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:24.983593 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc00cea-a69e-4e48-90ef-c7a147696db8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcca0367-cdxzf\" (UID: \"1bc00cea-a69e-4e48-90ef-c7a147696db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcca0367-cdxzf" Oct 30 22:08:25.039732 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:25.039655 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q27hx\" (UniqueName: \"kubernetes.io/projected/1bc00cea-a69e-4e48-90ef-c7a147696db8-kube-api-access-q27hx\") pod \"maas-api-revoke-sub-bcca0367-cdxzf\" (UID: \"1bc00cea-a69e-4e48-90ef-c7a147696db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcca0367-cdxzf" Oct 30 22:08:25.092835 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:25.092794 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcca0367-cdxzf" Oct 30 22:08:25.472274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:25.472014 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcca0367-cdxzf"] Oct 30 22:08:25.478489 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:08:25.478457 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc00cea_a69e_4e48_90ef_c7a147696db8.slice/crio-0f38208ebce623bceb4e046559cd62d44af6072bc4df32e621e4ed10e7003574 WatchSource:0}: Error finding container 0f38208ebce623bceb4e046559cd62d44af6072bc4df32e621e4ed10e7003574: Status 404 returned error can't find the container with id 0f38208ebce623bceb4e046559cd62d44af6072bc4df32e621e4ed10e7003574 Oct 30 22:08:26.290013 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:26.289919 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcca0367-cdxzf" event={"ID":"1bc00cea-a69e-4e48-90ef-c7a147696db8","Type":"ContainerStarted","Data":"880ad40d0954770d4886c010099dff90eb6baff768198034546dbc576fd9e1b1"} Oct 30 22:08:26.290013 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:26.289975 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcca0367-cdxzf" event={"ID":"1bc00cea-a69e-4e48-90ef-c7a147696db8","Type":"ContainerStarted","Data":"0f38208ebce623bceb4e046559cd62d44af6072bc4df32e621e4ed10e7003574"} Oct 30 22:08:26.401588 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:26.401540 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcca0367-cdxzf" podStartSLOduration=2.401525351 podStartE2EDuration="2.401525351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:26.400353771 +0000 UTC m=+1108.952530071" watchObservedRunningTime="2026-10-30 22:08:26.401525351 +0000 UTC m=+1108.953701651" Oct 30 22:08:27.296948 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:27.296911 6006 generic.go:356] "Generic (PLEG): container finished" podID="1bc00cea-a69e-4e48-90ef-c7a147696db8" containerID="880ad40d0954770d4886c010099dff90eb6baff768198034546dbc576fd9e1b1" exitCode=0 Oct 30 22:08:27.297411 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:27.296975 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcca0367-cdxzf" event={"ID":"1bc00cea-a69e-4e48-90ef-c7a147696db8","Type":"ContainerDied","Data":"880ad40d0954770d4886c010099dff90eb6baff768198034546dbc576fd9e1b1"} Oct 30 22:08:27.931144 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:08:27.931102 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:08:28.450153 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:28.450126 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcca0367-cdxzf" Oct 30 22:08:28.522330 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:28.522275 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bc00cea-a69e-4e48-90ef-c7a147696db8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc00cea-a69e-4e48-90ef-c7a147696db8-maas-api-openshift-ca\") pod \"1bc00cea-a69e-4e48-90ef-c7a147696db8\" (UID: \"1bc00cea-a69e-4e48-90ef-c7a147696db8\") " Oct 30 22:08:28.522527 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:28.522373 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bc00cea-a69e-4e48-90ef-c7a147696db8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc00cea-a69e-4e48-90ef-c7a147696db8-maas-api-xks-ca\") pod \"1bc00cea-a69e-4e48-90ef-c7a147696db8\" (UID: \"1bc00cea-a69e-4e48-90ef-c7a147696db8\") " Oct 30 22:08:28.522588 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:28.522525 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bc00cea-a69e-4e48-90ef-c7a147696db8-kube-api-access-q27hx\" (UniqueName: \"kubernetes.io/projected/1bc00cea-a69e-4e48-90ef-c7a147696db8-kube-api-access-q27hx\") pod \"1bc00cea-a69e-4e48-90ef-c7a147696db8\" (UID: \"1bc00cea-a69e-4e48-90ef-c7a147696db8\") " Oct 30 22:08:28.523375 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:28.523172 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bc00cea-a69e-4e48-90ef-c7a147696db8-maas-api-openshift-ca" pod "1bc00cea-a69e-4e48-90ef-c7a147696db8" (UID: "1bc00cea-a69e-4e48-90ef-c7a147696db8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:28.524439 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:28.524400 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc00cea-a69e-4e48-90ef-c7a147696db8-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:28.525531 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:28.525492 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bc00cea-a69e-4e48-90ef-c7a147696db8-maas-api-xks-ca" pod "1bc00cea-a69e-4e48-90ef-c7a147696db8" (UID: "1bc00cea-a69e-4e48-90ef-c7a147696db8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:28.525654 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:28.525635 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc00cea-a69e-4e48-90ef-c7a147696db8-kube-api-access-q27hx" pod "1bc00cea-a69e-4e48-90ef-c7a147696db8" (UID: "1bc00cea-a69e-4e48-90ef-c7a147696db8"). InnerVolumeSpecName "kube-api-access-q27hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:28.625827 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:28.625712 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q27hx\" (UniqueName: \"kubernetes.io/projected/1bc00cea-a69e-4e48-90ef-c7a147696db8-kube-api-access-q27hx\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:28.625827 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:28.625760 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc00cea-a69e-4e48-90ef-c7a147696db8-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:29.310697 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.310664 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcca0367-cdxzf" Oct 30 22:08:29.310883 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.310664 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcca0367-cdxzf" event={"ID":"1bc00cea-a69e-4e48-90ef-c7a147696db8","Type":"ContainerDied","Data":"0f38208ebce623bceb4e046559cd62d44af6072bc4df32e621e4ed10e7003574"} Oct 30 22:08:29.310883 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.310826 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f38208ebce623bceb4e046559cd62d44af6072bc4df32e621e4ed10e7003574" Oct 30 22:08:29.728685 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.728580 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt"] Oct 30 22:08:29.729380 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.729356 6006 cpu_manager.go:414] "containerMap: removing container" podUID="1bc00cea-a69e-4e48-90ef-c7a147696db8" containerName="revoke-keys" Oct 30 22:08:29.729474 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.729380 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bc00cea-a69e-4e48-90ef-c7a147696db8" containerName="revoke-keys" Oct 30 22:08:29.729552 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.729539 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bc00cea-a69e-4e48-90ef-c7a147696db8" containerName="revoke-keys" Oct 30 22:08:29.733703 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.733679 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:29.838939 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.838903 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt"] Oct 30 22:08:29.841737 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.841699 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istiod-ca-cert\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:29.842058 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.842027 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-podinfo\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:29.842190 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.842071 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-token\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:29.842190 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.842102 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-credential-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:29.842190 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.842129 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-data\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:29.842190 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.842171 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-workload-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:29.842490 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.842196 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg2n8\" (UniqueName: \"kubernetes.io/projected/34b11b2f-2eb6-4313-9a00-74be3c58fecd-kube-api-access-hg2n8\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:29.842490 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.842273 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-workload-certs\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:29.842490 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.842296 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-envoy\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:29.872822 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.872790 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-779f32-openshift-default-dockercfg-lp8gt\"" Oct 30 22:08:29.943476 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.943434 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-workload-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:29.943673 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.943483 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg2n8\" (UniqueName: \"kubernetes.io/projected/34b11b2f-2eb6-4313-9a00-74be3c58fecd-kube-api-access-hg2n8\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:29.943673 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.943547 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-workload-certs\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:29.943673 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.943569 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-envoy\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:29.943847 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.943679 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istiod-ca-cert\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:29.943847 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.943730 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-podinfo\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:29.943847 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.943757 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-token\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:29.943847 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.943787 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-credential-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:29.943847 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.943812 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-data\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:29.944101 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.943875 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-workload-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:29.944244 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.944222 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-data\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:29.944363 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.944329 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-credential-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:29.944514 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.944487 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-workload-certs\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:29.944821 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.944797 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istiod-ca-cert\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:29.946712 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.946685 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-envoy\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:29.946852 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:29.946836 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-podinfo\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:30.002443 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.002351 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-token\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:30.002615 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.002578 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg2n8\" (UniqueName: \"kubernetes.io/projected/34b11b2f-2eb6-4313-9a00-74be3c58fecd-kube-api-access-hg2n8\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:30.048636 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.048602 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:30.487269 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.486756 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt"] Oct 30 22:08:30.509860 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:08:30.507282 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b11b2f_2eb6_4313_9a00_74be3c58fecd.slice/crio-51802e15e4e95d9356c74dfe3c45a423ef0b3c07efc7d83bb0b5e1eac23fe131 WatchSource:0}: Error finding container 51802e15e4e95d9356c74dfe3c45a423ef0b3c07efc7d83bb0b5e1eac23fe131: Status 404 returned error can't find the container with id 51802e15e4e95d9356c74dfe3c45a423ef0b3c07efc7d83bb0b5e1eac23fe131 Oct 30 22:08:30.523639 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.515974 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:08:30.523639 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.516114 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:08:30.523639 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.516183 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:08:30.699338 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.699299 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt"] Oct 30 22:08:30.708612 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.708563 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:30.761229 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.761143 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-7bb1c2-openshift-default-dockercfg-9dlcb\"" Oct 30 22:08:30.761975 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.761944 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt"] Oct 30 22:08:30.865712 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.865647 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-workload-certs\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:30.865908 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.865733 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-envoy\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:30.865908 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.865841 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-podinfo\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:30.865908 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.865866 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-token\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:30.866214 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.865910 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf391b9e-a66a-4b60-9c45-8025b5155829-istiod-ca-cert\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:30.866214 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.865979 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-workload-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:30.866214 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.866013 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-credential-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:30.866214 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.866036 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scs62\" (UniqueName: \"kubernetes.io/projected/bf391b9e-a66a-4b60-9c45-8025b5155829-kube-api-access-scs62\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:30.866214 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.866076 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-data\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:30.967488 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.967448 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-podinfo\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:30.967488 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.967495 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-token\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:30.967705 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.967537 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf391b9e-a66a-4b60-9c45-8025b5155829-istiod-ca-cert\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:30.967705 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.967607 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-workload-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:30.967705 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.967648 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-credential-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:30.967705 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.967672 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scs62\" (UniqueName: \"kubernetes.io/projected/bf391b9e-a66a-4b60-9c45-8025b5155829-kube-api-access-scs62\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:30.967835 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.967715 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-data\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:30.967835 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.967774 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-workload-certs\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:30.967835 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.967814 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-envoy\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:30.969604 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.968455 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-credential-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:30.969604 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.968915 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf391b9e-a66a-4b60-9c45-8025b5155829-istiod-ca-cert\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:30.969604 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.969266 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-data\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:30.969604 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.969533 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-workload-certs\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:30.971424 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.969650 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-workload-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:30.971424 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.970564 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-envoy\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:30.973009 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:30.972987 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-podinfo\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:31.029492 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:31.029405 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-token\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:31.037125 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:31.037081 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scs62\" (UniqueName: \"kubernetes.io/projected/bf391b9e-a66a-4b60-9c45-8025b5155829-kube-api-access-scs62\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:31.131091 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:31.131047 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:31.349456 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:31.349411 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" event={"ID":"34b11b2f-2eb6-4313-9a00-74be3c58fecd","Type":"ContainerStarted","Data":"926b17f945f88db77ff307ac8264332a147db9020a96ccb8eb7508ca3bafa007"} Oct 30 22:08:31.349610 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:31.349470 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" event={"ID":"34b11b2f-2eb6-4313-9a00-74be3c58fecd","Type":"ContainerStarted","Data":"51802e15e4e95d9356c74dfe3c45a423ef0b3c07efc7d83bb0b5e1eac23fe131"} Oct 30 22:08:31.499282 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:31.498815 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" podStartSLOduration=2.498792907 podStartE2EDuration="2.498792907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:31.493160038 +0000 UTC m=+1114.045336339" watchObservedRunningTime="2026-10-30 22:08:31.498792907 +0000 UTC m=+1114.050969208" Oct 30 22:08:31.579483 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:31.579447 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt"] Oct 30 22:08:31.581800 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:08:31.581766 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf391b9e_a66a_4b60_9c45_8025b5155829.slice/crio-65741f23f5cc5a765238efca3abd00fbaf97898579d6a6a6ad6c024480a4b5f9 WatchSource:0}: Error finding container 65741f23f5cc5a765238efca3abd00fbaf97898579d6a6a6ad6c024480a4b5f9: Status 404 returned error can't find the container with id 65741f23f5cc5a765238efca3abd00fbaf97898579d6a6a6ad6c024480a4b5f9 Oct 30 22:08:31.584828 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:31.584747 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:08:31.584976 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:31.584883 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:08:31.584976 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:31.584927 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:08:32.050413 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:32.050358 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:32.050892 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:32.050491 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:08:32.050892 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:32.050549 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" podUID="34b11b2f-2eb6-4313-9a00-74be3c58fecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 22:08:32.397945 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:32.397844 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" event={"ID":"bf391b9e-a66a-4b60-9c45-8025b5155829","Type":"ContainerStarted","Data":"a15439709ea391e6878e0220c6ba1ea463681ea21e3a3372672298292b75475e"} Oct 30 22:08:32.397945 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:32.397944 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" event={"ID":"bf391b9e-a66a-4b60-9c45-8025b5155829","Type":"ContainerStarted","Data":"65741f23f5cc5a765238efca3abd00fbaf97898579d6a6a6ad6c024480a4b5f9"} Oct 30 22:08:32.518345 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:32.518273 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" podStartSLOduration=2.518232936 podStartE2EDuration="2.518232936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:32.515856028 +0000 UTC m=+1115.068032332" watchObservedRunningTime="2026-10-30 22:08:32.518232936 +0000 UTC m=+1115.070409239" Oct 30 22:08:33.056275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:33.055733 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:08:33.056275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:33.055806 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" podUID="34b11b2f-2eb6-4313-9a00-74be3c58fecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 22:08:33.145273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:33.144555 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:33.158275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:33.146359 6006 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:08:33.158275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:33.146421 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" podUID="bf391b9e-a66a-4b60-9c45-8025b5155829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 22:08:34.060274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:34.059707 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:08:34.060274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:34.059777 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" podUID="34b11b2f-2eb6-4313-9a00-74be3c58fecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 22:08:34.138733 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:34.138688 6006 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:08:34.138995 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:34.138965 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" podUID="bf391b9e-a66a-4b60-9c45-8025b5155829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 22:08:34.248423 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:34.248361 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:34.438332 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:34.437426 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:34.465794 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:34.465356 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:34.763365 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:34.762331 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:35.039465 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:35.039357 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:35.049739 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:35.049695 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:08:35.050067 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:35.050027 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" podUID="34b11b2f-2eb6-4313-9a00-74be3c58fecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 22:08:35.134807 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:35.134769 6006 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:08:35.135483 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:35.135438 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" podUID="bf391b9e-a66a-4b60-9c45-8025b5155829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 22:08:36.051843 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:36.051804 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:08:36.052170 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:36.052140 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" podUID="34b11b2f-2eb6-4313-9a00-74be3c58fecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 22:08:36.133785 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:36.133747 6006 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:08:36.134057 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:36.134032 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" podUID="bf391b9e-a66a-4b60-9c45-8025b5155829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 22:08:37.050273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:37.049704 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:08:37.050273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:37.049771 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" podUID="34b11b2f-2eb6-4313-9a00-74be3c58fecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 22:08:37.143274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:37.136667 6006 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:08:37.143274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:37.136729 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" podUID="bf391b9e-a66a-4b60-9c45-8025b5155829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 22:08:38.133293 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:38.132805 6006 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:08:38.133293 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:38.132878 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" podUID="bf391b9e-a66a-4b60-9c45-8025b5155829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 22:08:38.142890 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:38.142854 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:38.145724 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:38.145702 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:38.146990 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:38.146970 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:08:39.150747 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:39.150710 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:39.151395 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:39.151266 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:39.153435 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:39.153413 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:08:46.567724 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:46.567687 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9a3e41c-tmgdw"] Oct 30 22:08:46.571800 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:46.571776 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9a3e41c-tmgdw" Oct 30 22:08:46.623493 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:46.623463 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:08:46.627610 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:46.627494 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9a3e41c-tmgdw"] Oct 30 22:08:46.648225 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:46.648190 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f11701d-ecc8-4627-bb0e-6e99a8efebea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9a3e41c-tmgdw\" (UID: \"7f11701d-ecc8-4627-bb0e-6e99a8efebea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9a3e41c-tmgdw" Oct 30 22:08:46.648438 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:46.648283 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74jhb\" (UniqueName: \"kubernetes.io/projected/7f11701d-ecc8-4627-bb0e-6e99a8efebea-kube-api-access-74jhb\") pod \"maas-api-revoke-sub-a9a3e41c-tmgdw\" (UID: \"7f11701d-ecc8-4627-bb0e-6e99a8efebea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9a3e41c-tmgdw" Oct 30 22:08:46.648438 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:46.648354 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f11701d-ecc8-4627-bb0e-6e99a8efebea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9a3e41c-tmgdw\" (UID: \"7f11701d-ecc8-4627-bb0e-6e99a8efebea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9a3e41c-tmgdw" Oct 30 22:08:46.670155 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:46.670122 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:08:46.749582 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:46.749543 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f11701d-ecc8-4627-bb0e-6e99a8efebea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9a3e41c-tmgdw\" (UID: \"7f11701d-ecc8-4627-bb0e-6e99a8efebea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9a3e41c-tmgdw" Oct 30 22:08:46.749775 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:46.749608 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74jhb\" (UniqueName: \"kubernetes.io/projected/7f11701d-ecc8-4627-bb0e-6e99a8efebea-kube-api-access-74jhb\") pod \"maas-api-revoke-sub-a9a3e41c-tmgdw\" (UID: \"7f11701d-ecc8-4627-bb0e-6e99a8efebea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9a3e41c-tmgdw" Oct 30 22:08:46.749775 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:46.749642 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f11701d-ecc8-4627-bb0e-6e99a8efebea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9a3e41c-tmgdw\" (UID: \"7f11701d-ecc8-4627-bb0e-6e99a8efebea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9a3e41c-tmgdw" Oct 30 22:08:46.750388 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:46.750366 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f11701d-ecc8-4627-bb0e-6e99a8efebea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9a3e41c-tmgdw\" (UID: \"7f11701d-ecc8-4627-bb0e-6e99a8efebea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9a3e41c-tmgdw" Oct 30 22:08:46.752053 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:46.752030 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f11701d-ecc8-4627-bb0e-6e99a8efebea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9a3e41c-tmgdw\" (UID: \"7f11701d-ecc8-4627-bb0e-6e99a8efebea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9a3e41c-tmgdw" Oct 30 22:08:46.808033 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:46.808001 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74jhb\" (UniqueName: \"kubernetes.io/projected/7f11701d-ecc8-4627-bb0e-6e99a8efebea-kube-api-access-74jhb\") pod \"maas-api-revoke-sub-a9a3e41c-tmgdw\" (UID: \"7f11701d-ecc8-4627-bb0e-6e99a8efebea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9a3e41c-tmgdw" Oct 30 22:08:46.885848 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:46.885758 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9a3e41c-tmgdw" Oct 30 22:08:47.257675 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:47.257638 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9a3e41c-tmgdw"] Oct 30 22:08:47.259069 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:08:47.259037 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f11701d_ecc8_4627_bb0e_6e99a8efebea.slice/crio-76cf72f251b2d5ae0ff3cd45c3d080bdf2062bc4243d87d0ed0d002c6a73fd0f WatchSource:0}: Error finding container 76cf72f251b2d5ae0ff3cd45c3d080bdf2062bc4243d87d0ed0d002c6a73fd0f: Status 404 returned error can't find the container with id 76cf72f251b2d5ae0ff3cd45c3d080bdf2062bc4243d87d0ed0d002c6a73fd0f Oct 30 22:08:47.551717 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:47.551672 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9a3e41c-tmgdw" event={"ID":"7f11701d-ecc8-4627-bb0e-6e99a8efebea","Type":"ContainerStarted","Data":"76cf72f251b2d5ae0ff3cd45c3d080bdf2062bc4243d87d0ed0d002c6a73fd0f"} Oct 30 22:08:48.560280 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:48.560214 6006 generic.go:356] "Generic (PLEG): container finished" podID="7f11701d-ecc8-4627-bb0e-6e99a8efebea" containerID="5ae4cbb8c76b125aff9c3939d66c3100f4eb6644a86fdfc9328363ca1666090b" exitCode=0 Oct 30 22:08:48.560280 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:48.560271 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9a3e41c-tmgdw" event={"ID":"7f11701d-ecc8-4627-bb0e-6e99a8efebea","Type":"ContainerDied","Data":"5ae4cbb8c76b125aff9c3939d66c3100f4eb6644a86fdfc9328363ca1666090b"} Oct 30 22:08:49.725585 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:49.725549 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9a3e41c-tmgdw" Oct 30 22:08:49.785901 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:49.785863 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f11701d-ecc8-4627-bb0e-6e99a8efebea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f11701d-ecc8-4627-bb0e-6e99a8efebea-maas-api-openshift-ca\") pod \"7f11701d-ecc8-4627-bb0e-6e99a8efebea\" (UID: \"7f11701d-ecc8-4627-bb0e-6e99a8efebea\") " Oct 30 22:08:49.786092 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:49.785971 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f11701d-ecc8-4627-bb0e-6e99a8efebea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f11701d-ecc8-4627-bb0e-6e99a8efebea-maas-api-xks-ca\") pod \"7f11701d-ecc8-4627-bb0e-6e99a8efebea\" (UID: \"7f11701d-ecc8-4627-bb0e-6e99a8efebea\") " Oct 30 22:08:49.786092 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:49.786057 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f11701d-ecc8-4627-bb0e-6e99a8efebea-kube-api-access-74jhb\" (UniqueName: \"kubernetes.io/projected/7f11701d-ecc8-4627-bb0e-6e99a8efebea-kube-api-access-74jhb\") pod \"7f11701d-ecc8-4627-bb0e-6e99a8efebea\" (UID: \"7f11701d-ecc8-4627-bb0e-6e99a8efebea\") " Oct 30 22:08:49.786264 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:49.786226 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f11701d-ecc8-4627-bb0e-6e99a8efebea-maas-api-openshift-ca" pod "7f11701d-ecc8-4627-bb0e-6e99a8efebea" (UID: "7f11701d-ecc8-4627-bb0e-6e99a8efebea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:49.786540 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:49.786515 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f11701d-ecc8-4627-bb0e-6e99a8efebea-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:49.788183 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:49.788160 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f11701d-ecc8-4627-bb0e-6e99a8efebea-maas-api-xks-ca" pod "7f11701d-ecc8-4627-bb0e-6e99a8efebea" (UID: "7f11701d-ecc8-4627-bb0e-6e99a8efebea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:49.788308 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:49.788217 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f11701d-ecc8-4627-bb0e-6e99a8efebea-kube-api-access-74jhb" pod "7f11701d-ecc8-4627-bb0e-6e99a8efebea" (UID: "7f11701d-ecc8-4627-bb0e-6e99a8efebea"). InnerVolumeSpecName "kube-api-access-74jhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:49.887433 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:49.887327 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74jhb\" (UniqueName: \"kubernetes.io/projected/7f11701d-ecc8-4627-bb0e-6e99a8efebea-kube-api-access-74jhb\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:49.887433 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:49.887365 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f11701d-ecc8-4627-bb0e-6e99a8efebea-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:50.573562 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:50.573531 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9a3e41c-tmgdw" Oct 30 22:08:50.573751 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:50.573561 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9a3e41c-tmgdw" event={"ID":"7f11701d-ecc8-4627-bb0e-6e99a8efebea","Type":"ContainerDied","Data":"76cf72f251b2d5ae0ff3cd45c3d080bdf2062bc4243d87d0ed0d002c6a73fd0f"} Oct 30 22:08:50.573751 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:50.573646 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76cf72f251b2d5ae0ff3cd45c3d080bdf2062bc4243d87d0ed0d002c6a73fd0f" Oct 30 22:08:50.795241 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:50.795202 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-837c347f-q62v4"] Oct 30 22:08:50.795806 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:50.795788 6006 cpu_manager.go:414] "containerMap: removing container" podUID="7f11701d-ecc8-4627-bb0e-6e99a8efebea" containerName="revoke-keys" Oct 30 22:08:50.795806 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:50.795806 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f11701d-ecc8-4627-bb0e-6e99a8efebea" containerName="revoke-keys" Oct 30 22:08:50.795949 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:50.795886 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f11701d-ecc8-4627-bb0e-6e99a8efebea" containerName="revoke-keys" Oct 30 22:08:50.805794 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:50.805761 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837c347f-q62v4" Oct 30 22:08:50.857887 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:50.857788 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-837c347f-q62v4"] Oct 30 22:08:50.899453 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:50.899419 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7587b568-cf56-45a8-97f7-52d98edf7b17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-837c347f-q62v4\" (UID: \"7587b568-cf56-45a8-97f7-52d98edf7b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837c347f-q62v4" Oct 30 22:08:50.899645 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:50.899528 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np8kf\" (UniqueName: \"kubernetes.io/projected/7587b568-cf56-45a8-97f7-52d98edf7b17-kube-api-access-np8kf\") pod \"maas-api-revoke-sub-837c347f-q62v4\" (UID: \"7587b568-cf56-45a8-97f7-52d98edf7b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837c347f-q62v4" Oct 30 22:08:50.899688 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:50.899660 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7587b568-cf56-45a8-97f7-52d98edf7b17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-837c347f-q62v4\" (UID: \"7587b568-cf56-45a8-97f7-52d98edf7b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837c347f-q62v4" Oct 30 22:08:50.901217 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:50.901189 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:08:50.901217 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:50.901190 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:08:51.000581 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:51.000545 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7587b568-cf56-45a8-97f7-52d98edf7b17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-837c347f-q62v4\" (UID: \"7587b568-cf56-45a8-97f7-52d98edf7b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837c347f-q62v4" Oct 30 22:08:51.000800 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:51.000664 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np8kf\" (UniqueName: \"kubernetes.io/projected/7587b568-cf56-45a8-97f7-52d98edf7b17-kube-api-access-np8kf\") pod \"maas-api-revoke-sub-837c347f-q62v4\" (UID: \"7587b568-cf56-45a8-97f7-52d98edf7b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837c347f-q62v4" Oct 30 22:08:51.000800 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:51.000722 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7587b568-cf56-45a8-97f7-52d98edf7b17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-837c347f-q62v4\" (UID: \"7587b568-cf56-45a8-97f7-52d98edf7b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837c347f-q62v4" Oct 30 22:08:51.001364 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:51.001336 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7587b568-cf56-45a8-97f7-52d98edf7b17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-837c347f-q62v4\" (UID: \"7587b568-cf56-45a8-97f7-52d98edf7b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837c347f-q62v4" Oct 30 22:08:51.003116 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:51.003095 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7587b568-cf56-45a8-97f7-52d98edf7b17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-837c347f-q62v4\" (UID: \"7587b568-cf56-45a8-97f7-52d98edf7b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837c347f-q62v4" Oct 30 22:08:51.059720 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:51.059685 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np8kf\" (UniqueName: \"kubernetes.io/projected/7587b568-cf56-45a8-97f7-52d98edf7b17-kube-api-access-np8kf\") pod \"maas-api-revoke-sub-837c347f-q62v4\" (UID: \"7587b568-cf56-45a8-97f7-52d98edf7b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837c347f-q62v4" Oct 30 22:08:51.123397 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:51.123298 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837c347f-q62v4" Oct 30 22:08:51.503575 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:51.503543 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-837c347f-q62v4"] Oct 30 22:08:51.505713 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:08:51.505675 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7587b568_cf56_45a8_97f7_52d98edf7b17.slice/crio-ce8f74244b3c93f37a2ad2a013afc19c663c97d66e67ac54ff5fb87634032518 WatchSource:0}: Error finding container ce8f74244b3c93f37a2ad2a013afc19c663c97d66e67ac54ff5fb87634032518: Status 404 returned error can't find the container with id ce8f74244b3c93f37a2ad2a013afc19c663c97d66e67ac54ff5fb87634032518 Oct 30 22:08:51.588715 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:51.588669 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837c347f-q62v4" event={"ID":"7587b568-cf56-45a8-97f7-52d98edf7b17","Type":"ContainerStarted","Data":"ce8f74244b3c93f37a2ad2a013afc19c663c97d66e67ac54ff5fb87634032518"} Oct 30 22:08:52.597377 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:52.597339 6006 generic.go:356] "Generic (PLEG): container finished" podID="7587b568-cf56-45a8-97f7-52d98edf7b17" containerID="f6ae4f6a8e784e01380f575fb36dafef94dc334d941debe33d463155d6473e05" exitCode=0 Oct 30 22:08:52.597775 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:52.597425 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837c347f-q62v4" event={"ID":"7587b568-cf56-45a8-97f7-52d98edf7b17","Type":"ContainerDied","Data":"f6ae4f6a8e784e01380f575fb36dafef94dc334d941debe33d463155d6473e05"} Oct 30 22:08:53.780490 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:53.780463 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837c347f-q62v4" Oct 30 22:08:53.810034 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:53.809996 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:08:53.841670 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:53.841639 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7587b568-cf56-45a8-97f7-52d98edf7b17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7587b568-cf56-45a8-97f7-52d98edf7b17-maas-api-openshift-ca\") pod \"7587b568-cf56-45a8-97f7-52d98edf7b17\" (UID: \"7587b568-cf56-45a8-97f7-52d98edf7b17\") " Oct 30 22:08:53.841846 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:53.841751 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7587b568-cf56-45a8-97f7-52d98edf7b17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7587b568-cf56-45a8-97f7-52d98edf7b17-maas-api-xks-ca\") pod \"7587b568-cf56-45a8-97f7-52d98edf7b17\" (UID: \"7587b568-cf56-45a8-97f7-52d98edf7b17\") " Oct 30 22:08:53.841846 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:53.841776 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7587b568-cf56-45a8-97f7-52d98edf7b17-kube-api-access-np8kf\" (UniqueName: \"kubernetes.io/projected/7587b568-cf56-45a8-97f7-52d98edf7b17-kube-api-access-np8kf\") pod \"7587b568-cf56-45a8-97f7-52d98edf7b17\" (UID: \"7587b568-cf56-45a8-97f7-52d98edf7b17\") " Oct 30 22:08:53.842087 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:53.842058 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7587b568-cf56-45a8-97f7-52d98edf7b17-maas-api-openshift-ca" pod "7587b568-cf56-45a8-97f7-52d98edf7b17" (UID: "7587b568-cf56-45a8-97f7-52d98edf7b17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:53.842206 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:53.842185 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7587b568-cf56-45a8-97f7-52d98edf7b17-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:53.843917 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:53.843892 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7587b568-cf56-45a8-97f7-52d98edf7b17-maas-api-xks-ca" pod "7587b568-cf56-45a8-97f7-52d98edf7b17" (UID: "7587b568-cf56-45a8-97f7-52d98edf7b17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:53.844041 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:53.843968 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7587b568-cf56-45a8-97f7-52d98edf7b17-kube-api-access-np8kf" pod "7587b568-cf56-45a8-97f7-52d98edf7b17" (UID: "7587b568-cf56-45a8-97f7-52d98edf7b17"). InnerVolumeSpecName "kube-api-access-np8kf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:53.942949 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:53.942856 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7587b568-cf56-45a8-97f7-52d98edf7b17-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:53.942949 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:53.942895 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np8kf\" (UniqueName: \"kubernetes.io/projected/7587b568-cf56-45a8-97f7-52d98edf7b17-kube-api-access-np8kf\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:54.525441 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.525383 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9291b53a-gdbsm"] Oct 30 22:08:54.526743 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.526705 6006 cpu_manager.go:414] "containerMap: removing container" podUID="7587b568-cf56-45a8-97f7-52d98edf7b17" containerName="revoke-keys" Oct 30 22:08:54.526910 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.526750 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7587b568-cf56-45a8-97f7-52d98edf7b17" containerName="revoke-keys" Oct 30 22:08:54.526995 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.526962 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="7587b568-cf56-45a8-97f7-52d98edf7b17" containerName="revoke-keys" Oct 30 22:08:54.551285 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.544504 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9291b53a-gdbsm" Oct 30 22:08:54.593297 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.585348 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9291b53a-gdbsm"] Oct 30 22:08:54.617788 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.617749 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837c347f-q62v4" Oct 30 22:08:54.618001 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.617758 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837c347f-q62v4" event={"ID":"7587b568-cf56-45a8-97f7-52d98edf7b17","Type":"ContainerDied","Data":"ce8f74244b3c93f37a2ad2a013afc19c663c97d66e67ac54ff5fb87634032518"} Oct 30 22:08:54.618001 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.617883 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce8f74244b3c93f37a2ad2a013afc19c663c97d66e67ac54ff5fb87634032518" Oct 30 22:08:54.633429 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.633396 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q"] Oct 30 22:08:54.641189 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.641161 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.652800 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.652763 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/023471a9-eb6d-410a-8fc9-4ee320f6f481-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9291b53a-gdbsm\" (UID: \"023471a9-eb6d-410a-8fc9-4ee320f6f481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9291b53a-gdbsm" Oct 30 22:08:54.652963 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.652822 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/023471a9-eb6d-410a-8fc9-4ee320f6f481-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9291b53a-gdbsm\" (UID: \"023471a9-eb6d-410a-8fc9-4ee320f6f481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9291b53a-gdbsm" Oct 30 22:08:54.653063 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.653044 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n56fx\" (UniqueName: \"kubernetes.io/projected/023471a9-eb6d-410a-8fc9-4ee320f6f481-kube-api-access-n56fx\") pod \"maas-api-revoke-sub-9291b53a-gdbsm\" (UID: \"023471a9-eb6d-410a-8fc9-4ee320f6f481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9291b53a-gdbsm" Oct 30 22:08:54.750270 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.747930 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q"] Oct 30 22:08:54.758529 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.755789 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39c47890-6d91-4569-8e7d-74114231e81c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.758529 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.755850 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39c47890-6d91-4569-8e7d-74114231e81c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.758529 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.757463 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/023471a9-eb6d-410a-8fc9-4ee320f6f481-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9291b53a-gdbsm\" (UID: \"023471a9-eb6d-410a-8fc9-4ee320f6f481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9291b53a-gdbsm" Oct 30 22:08:54.758529 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.757517 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.758529 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.757565 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/023471a9-eb6d-410a-8fc9-4ee320f6f481-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9291b53a-gdbsm\" (UID: \"023471a9-eb6d-410a-8fc9-4ee320f6f481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9291b53a-gdbsm" Oct 30 22:08:54.758529 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.757641 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.758529 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.757692 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.758529 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.757727 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.758529 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.757772 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39c47890-6d91-4569-8e7d-74114231e81c-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.758529 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.757806 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxgf2\" (UniqueName: \"kubernetes.io/projected/39c47890-6d91-4569-8e7d-74114231e81c-kube-api-access-qxgf2\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.758529 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.757854 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.758529 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.757945 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n56fx\" (UniqueName: \"kubernetes.io/projected/023471a9-eb6d-410a-8fc9-4ee320f6f481-kube-api-access-n56fx\") pod \"maas-api-revoke-sub-9291b53a-gdbsm\" (UID: \"023471a9-eb6d-410a-8fc9-4ee320f6f481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9291b53a-gdbsm" Oct 30 22:08:54.760082 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.759264 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/023471a9-eb6d-410a-8fc9-4ee320f6f481-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9291b53a-gdbsm\" (UID: \"023471a9-eb6d-410a-8fc9-4ee320f6f481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9291b53a-gdbsm" Oct 30 22:08:54.768767 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.768736 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/023471a9-eb6d-410a-8fc9-4ee320f6f481-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9291b53a-gdbsm\" (UID: \"023471a9-eb6d-410a-8fc9-4ee320f6f481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9291b53a-gdbsm" Oct 30 22:08:54.823441 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.823393 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n56fx\" (UniqueName: \"kubernetes.io/projected/023471a9-eb6d-410a-8fc9-4ee320f6f481-kube-api-access-n56fx\") pod \"maas-api-revoke-sub-9291b53a-gdbsm\" (UID: \"023471a9-eb6d-410a-8fc9-4ee320f6f481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9291b53a-gdbsm" Oct 30 22:08:54.863849 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.858834 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39c47890-6d91-4569-8e7d-74114231e81c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.863849 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.858889 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39c47890-6d91-4569-8e7d-74114231e81c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.863849 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.858936 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.863849 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.859016 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.863849 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.859060 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.863849 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.859091 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.863849 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.859133 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39c47890-6d91-4569-8e7d-74114231e81c-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.863849 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.859161 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxgf2\" (UniqueName: \"kubernetes.io/projected/39c47890-6d91-4569-8e7d-74114231e81c-kube-api-access-qxgf2\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.863849 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.859205 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.863849 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.859938 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.863849 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.861013 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.863849 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.861496 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.863849 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.861802 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.863849 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.862210 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39c47890-6d91-4569-8e7d-74114231e81c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.881283 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.876993 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9291b53a-gdbsm" Oct 30 22:08:54.891742 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.889864 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39c47890-6d91-4569-8e7d-74114231e81c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.916968 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.909838 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.930962 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.930897 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39c47890-6d91-4569-8e7d-74114231e81c-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.947003 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.946954 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxgf2\" (UniqueName: \"kubernetes.io/projected/39c47890-6d91-4569-8e7d-74114231e81c-kube-api-access-qxgf2\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s5x8q\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:54.959394 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:54.959000 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:55.308195 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:55.308151 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9291b53a-gdbsm"] Oct 30 22:08:55.451282 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:55.448427 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q"] Oct 30 22:08:55.525303 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:55.515638 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:08:55.528943 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:55.528896 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:08:55.529210 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:55.529186 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:08:55.695904 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:55.695860 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9291b53a-gdbsm" event={"ID":"023471a9-eb6d-410a-8fc9-4ee320f6f481","Type":"ContainerStarted","Data":"c41ac77f90995fcddff09bd22c97c92a8dedab2510f48d7e219d27cfc6a2d373"} Oct 30 22:08:55.698582 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:55.698530 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" event={"ID":"39c47890-6d91-4569-8e7d-74114231e81c","Type":"ContainerStarted","Data":"7424f8687c5506f8ef0c8690acea0d47cbaf5d62b04d01268d746635199a14d9"} Oct 30 22:08:56.710502 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:56.710463 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" event={"ID":"39c47890-6d91-4569-8e7d-74114231e81c","Type":"ContainerStarted","Data":"c9a749d2e48b6e262628a8a9de7a14510583638c399e3f1571fb566d6a521776"} Oct 30 22:08:56.713115 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:56.713080 6006 generic.go:356] "Generic (PLEG): container finished" podID="023471a9-eb6d-410a-8fc9-4ee320f6f481" containerID="2ddd7e8901c56cc6903f69a9d4833b494c5023eb07b3b70c1fafeece84d5aa8d" exitCode=0 Oct 30 22:08:56.713298 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:56.713167 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9291b53a-gdbsm" event={"ID":"023471a9-eb6d-410a-8fc9-4ee320f6f481","Type":"ContainerDied","Data":"2ddd7e8901c56cc6903f69a9d4833b494c5023eb07b3b70c1fafeece84d5aa8d"} Oct 30 22:08:56.882878 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:56.882765 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" podStartSLOduration=2.882751419 podStartE2EDuration="2.882751419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:56.834887958 +0000 UTC m=+1139.387064260" watchObservedRunningTime="2026-10-30 22:08:56.882751419 +0000 UTC m=+1139.434927808" Oct 30 22:08:56.959792 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:56.959749 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:08:56.961798 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:56.961751 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-s5x8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:08:56.962129 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:56.961824 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" podUID="39c47890-6d91-4569-8e7d-74114231e81c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 22:08:57.935231 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:57.935191 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9291b53a-gdbsm" Oct 30 22:08:57.945420 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:08:57.945384 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:08:57.960268 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:57.960220 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-s5x8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:08:57.960445 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:57.960296 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" podUID="39c47890-6d91-4569-8e7d-74114231e81c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 22:08:58.044788 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:58.044747 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/023471a9-eb6d-410a-8fc9-4ee320f6f481-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/023471a9-eb6d-410a-8fc9-4ee320f6f481-maas-api-xks-ca\") pod \"023471a9-eb6d-410a-8fc9-4ee320f6f481\" (UID: \"023471a9-eb6d-410a-8fc9-4ee320f6f481\") " Oct 30 22:08:58.045006 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:58.044804 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/023471a9-eb6d-410a-8fc9-4ee320f6f481-kube-api-access-n56fx\" (UniqueName: \"kubernetes.io/projected/023471a9-eb6d-410a-8fc9-4ee320f6f481-kube-api-access-n56fx\") pod \"023471a9-eb6d-410a-8fc9-4ee320f6f481\" (UID: \"023471a9-eb6d-410a-8fc9-4ee320f6f481\") " Oct 30 22:08:58.045006 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:58.044926 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/023471a9-eb6d-410a-8fc9-4ee320f6f481-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/023471a9-eb6d-410a-8fc9-4ee320f6f481-maas-api-openshift-ca\") pod \"023471a9-eb6d-410a-8fc9-4ee320f6f481\" (UID: \"023471a9-eb6d-410a-8fc9-4ee320f6f481\") " Oct 30 22:08:58.045650 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:58.045619 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/023471a9-eb6d-410a-8fc9-4ee320f6f481-maas-api-openshift-ca" pod "023471a9-eb6d-410a-8fc9-4ee320f6f481" (UID: "023471a9-eb6d-410a-8fc9-4ee320f6f481"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:58.048955 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:58.048923 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/023471a9-eb6d-410a-8fc9-4ee320f6f481-kube-api-access-n56fx" pod "023471a9-eb6d-410a-8fc9-4ee320f6f481" (UID: "023471a9-eb6d-410a-8fc9-4ee320f6f481"). InnerVolumeSpecName "kube-api-access-n56fx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:58.049130 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:58.048927 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/023471a9-eb6d-410a-8fc9-4ee320f6f481-maas-api-xks-ca" pod "023471a9-eb6d-410a-8fc9-4ee320f6f481" (UID: "023471a9-eb6d-410a-8fc9-4ee320f6f481"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:58.146908 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:58.146874 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/023471a9-eb6d-410a-8fc9-4ee320f6f481-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:58.146908 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:58.146905 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n56fx\" (UniqueName: \"kubernetes.io/projected/023471a9-eb6d-410a-8fc9-4ee320f6f481-kube-api-access-n56fx\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:58.146908 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:58.146916 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/023471a9-eb6d-410a-8fc9-4ee320f6f481-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:58.745624 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:58.745306 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9291b53a-gdbsm" Oct 30 22:08:58.745624 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:58.745421 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9291b53a-gdbsm" event={"ID":"023471a9-eb6d-410a-8fc9-4ee320f6f481","Type":"ContainerDied","Data":"c41ac77f90995fcddff09bd22c97c92a8dedab2510f48d7e219d27cfc6a2d373"} Oct 30 22:08:58.745624 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:58.745562 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c41ac77f90995fcddff09bd22c97c92a8dedab2510f48d7e219d27cfc6a2d373" Oct 30 22:08:58.960002 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:58.959963 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-s5x8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:08:58.960410 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:58.960024 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" podUID="39c47890-6d91-4569-8e7d-74114231e81c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 22:08:59.960086 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:59.960049 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-s5x8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:08:59.960527 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:08:59.960109 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" podUID="39c47890-6d91-4569-8e7d-74114231e81c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 22:09:00.960685 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:00.960637 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-s5x8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:09:00.961206 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:00.960707 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" podUID="39c47890-6d91-4569-8e7d-74114231e81c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 22:09:01.959638 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:01.959604 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-s5x8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:09:01.959842 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:01.959664 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" podUID="39c47890-6d91-4569-8e7d-74114231e81c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 22:09:02.168265 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:02.168222 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c65ead47-qssdz"] Oct 30 22:09:02.169152 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:02.169119 6006 cpu_manager.go:414] "containerMap: removing container" podUID="023471a9-eb6d-410a-8fc9-4ee320f6f481" containerName="revoke-keys" Oct 30 22:09:02.169152 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:02.169145 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="023471a9-eb6d-410a-8fc9-4ee320f6f481" containerName="revoke-keys" Oct 30 22:09:02.169378 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:02.169283 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="023471a9-eb6d-410a-8fc9-4ee320f6f481" containerName="revoke-keys" Oct 30 22:09:02.173219 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:02.173196 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65ead47-qssdz" Oct 30 22:09:02.228027 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:02.227934 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c65ead47-qssdz"] Oct 30 22:09:02.264475 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:02.264435 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:09:02.264675 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:02.264482 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:09:02.295557 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:02.295521 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l4kj\" (UniqueName: \"kubernetes.io/projected/f71ebd22-a1d9-4f0c-b56c-8a98f7908255-kube-api-access-9l4kj\") pod \"maas-api-revoke-sub-c65ead47-qssdz\" (UID: \"f71ebd22-a1d9-4f0c-b56c-8a98f7908255\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65ead47-qssdz" Oct 30 22:09:02.295720 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:02.295573 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f71ebd22-a1d9-4f0c-b56c-8a98f7908255-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c65ead47-qssdz\" (UID: \"f71ebd22-a1d9-4f0c-b56c-8a98f7908255\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65ead47-qssdz" Oct 30 22:09:02.295758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:02.295725 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f71ebd22-a1d9-4f0c-b56c-8a98f7908255-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c65ead47-qssdz\" (UID: \"f71ebd22-a1d9-4f0c-b56c-8a98f7908255\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65ead47-qssdz" Oct 30 22:09:02.397015 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:02.396976 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f71ebd22-a1d9-4f0c-b56c-8a98f7908255-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c65ead47-qssdz\" (UID: \"f71ebd22-a1d9-4f0c-b56c-8a98f7908255\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65ead47-qssdz" Oct 30 22:09:02.397243 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:02.397122 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l4kj\" (UniqueName: \"kubernetes.io/projected/f71ebd22-a1d9-4f0c-b56c-8a98f7908255-kube-api-access-9l4kj\") pod \"maas-api-revoke-sub-c65ead47-qssdz\" (UID: \"f71ebd22-a1d9-4f0c-b56c-8a98f7908255\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65ead47-qssdz" Oct 30 22:09:02.397243 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:02.397172 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f71ebd22-a1d9-4f0c-b56c-8a98f7908255-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c65ead47-qssdz\" (UID: \"f71ebd22-a1d9-4f0c-b56c-8a98f7908255\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65ead47-qssdz" Oct 30 22:09:02.397870 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:02.397848 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f71ebd22-a1d9-4f0c-b56c-8a98f7908255-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c65ead47-qssdz\" (UID: \"f71ebd22-a1d9-4f0c-b56c-8a98f7908255\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65ead47-qssdz" Oct 30 22:09:02.399532 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:02.399506 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f71ebd22-a1d9-4f0c-b56c-8a98f7908255-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c65ead47-qssdz\" (UID: \"f71ebd22-a1d9-4f0c-b56c-8a98f7908255\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65ead47-qssdz" Oct 30 22:09:02.456236 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:02.456204 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l4kj\" (UniqueName: \"kubernetes.io/projected/f71ebd22-a1d9-4f0c-b56c-8a98f7908255-kube-api-access-9l4kj\") pod \"maas-api-revoke-sub-c65ead47-qssdz\" (UID: \"f71ebd22-a1d9-4f0c-b56c-8a98f7908255\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65ead47-qssdz" Oct 30 22:09:02.487162 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:02.487067 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65ead47-qssdz" Oct 30 22:09:02.860971 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:02.860943 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c65ead47-qssdz"] Oct 30 22:09:02.862763 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:09:02.862733 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71ebd22_a1d9_4f0c_b56c_8a98f7908255.slice/crio-830d71e2e7e7a56a140c7ca94b0df4b7489587f445a1159e2a948c6928f853ba WatchSource:0}: Error finding container 830d71e2e7e7a56a140c7ca94b0df4b7489587f445a1159e2a948c6928f853ba: Status 404 returned error can't find the container with id 830d71e2e7e7a56a140c7ca94b0df4b7489587f445a1159e2a948c6928f853ba Oct 30 22:09:02.960914 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:02.960880 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-s5x8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:09:02.961088 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:02.960934 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" podUID="39c47890-6d91-4569-8e7d-74114231e81c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 22:09:03.788850 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:03.788815 6006 generic.go:356] "Generic (PLEG): container finished" podID="f71ebd22-a1d9-4f0c-b56c-8a98f7908255" containerID="26579db3a73415884ae9cae4c6c84a10bdca4aad6bbf6847b0970ba33513c4ae" exitCode=0 Oct 30 22:09:03.789397 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:03.788877 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65ead47-qssdz" event={"ID":"f71ebd22-a1d9-4f0c-b56c-8a98f7908255","Type":"ContainerDied","Data":"26579db3a73415884ae9cae4c6c84a10bdca4aad6bbf6847b0970ba33513c4ae"} Oct 30 22:09:03.789397 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:03.788917 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65ead47-qssdz" event={"ID":"f71ebd22-a1d9-4f0c-b56c-8a98f7908255","Type":"ContainerStarted","Data":"830d71e2e7e7a56a140c7ca94b0df4b7489587f445a1159e2a948c6928f853ba"} Oct 30 22:09:03.960771 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:03.960680 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-s5x8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:09:03.960771 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:03.960741 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" podUID="39c47890-6d91-4569-8e7d-74114231e81c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 22:09:04.965476 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:04.965447 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:09:04.965842 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:04.965820 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:09:04.966722 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:04.966697 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:09:04.967741 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:04.967723 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65ead47-qssdz" Oct 30 22:09:05.029581 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:05.029544 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f71ebd22-a1d9-4f0c-b56c-8a98f7908255-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f71ebd22-a1d9-4f0c-b56c-8a98f7908255-maas-api-openshift-ca\") pod \"f71ebd22-a1d9-4f0c-b56c-8a98f7908255\" (UID: \"f71ebd22-a1d9-4f0c-b56c-8a98f7908255\") " Oct 30 22:09:05.029760 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:05.029642 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f71ebd22-a1d9-4f0c-b56c-8a98f7908255-kube-api-access-9l4kj\" (UniqueName: \"kubernetes.io/projected/f71ebd22-a1d9-4f0c-b56c-8a98f7908255-kube-api-access-9l4kj\") pod \"f71ebd22-a1d9-4f0c-b56c-8a98f7908255\" (UID: \"f71ebd22-a1d9-4f0c-b56c-8a98f7908255\") " Oct 30 22:09:05.029760 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:05.029677 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f71ebd22-a1d9-4f0c-b56c-8a98f7908255-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f71ebd22-a1d9-4f0c-b56c-8a98f7908255-maas-api-xks-ca\") pod \"f71ebd22-a1d9-4f0c-b56c-8a98f7908255\" (UID: \"f71ebd22-a1d9-4f0c-b56c-8a98f7908255\") " Oct 30 22:09:05.030181 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:05.029979 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f71ebd22-a1d9-4f0c-b56c-8a98f7908255-maas-api-openshift-ca" pod "f71ebd22-a1d9-4f0c-b56c-8a98f7908255" (UID: "f71ebd22-a1d9-4f0c-b56c-8a98f7908255"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:05.030181 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:05.030129 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f71ebd22-a1d9-4f0c-b56c-8a98f7908255-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:05.031939 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:05.031911 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f71ebd22-a1d9-4f0c-b56c-8a98f7908255-maas-api-xks-ca" pod "f71ebd22-a1d9-4f0c-b56c-8a98f7908255" (UID: "f71ebd22-a1d9-4f0c-b56c-8a98f7908255"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:05.032209 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:05.032184 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71ebd22-a1d9-4f0c-b56c-8a98f7908255-kube-api-access-9l4kj" pod "f71ebd22-a1d9-4f0c-b56c-8a98f7908255" (UID: "f71ebd22-a1d9-4f0c-b56c-8a98f7908255"). InnerVolumeSpecName "kube-api-access-9l4kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:05.131429 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:05.131327 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l4kj\" (UniqueName: \"kubernetes.io/projected/f71ebd22-a1d9-4f0c-b56c-8a98f7908255-kube-api-access-9l4kj\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:05.131429 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:05.131371 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f71ebd22-a1d9-4f0c-b56c-8a98f7908255-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:05.806103 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:05.806071 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65ead47-qssdz" Oct 30 22:09:05.806420 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:05.806110 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65ead47-qssdz" event={"ID":"f71ebd22-a1d9-4f0c-b56c-8a98f7908255","Type":"ContainerDied","Data":"830d71e2e7e7a56a140c7ca94b0df4b7489587f445a1159e2a948c6928f853ba"} Oct 30 22:09:05.806634 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:05.806616 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="830d71e2e7e7a56a140c7ca94b0df4b7489587f445a1159e2a948c6928f853ba" Oct 30 22:09:16.817819 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:16.817777 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p"] Oct 30 22:09:16.818346 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:16.818336 6006 cpu_manager.go:414] "containerMap: removing container" podUID="f71ebd22-a1d9-4f0c-b56c-8a98f7908255" containerName="revoke-keys" Oct 30 22:09:16.818396 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:16.818349 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f71ebd22-a1d9-4f0c-b56c-8a98f7908255" containerName="revoke-keys" Oct 30 22:09:16.818465 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:16.818455 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="f71ebd22-a1d9-4f0c-b56c-8a98f7908255" containerName="revoke-keys" Oct 30 22:09:16.821893 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:16.821872 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:16.871508 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:16.871473 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-65e59f-openshift-default-dockercfg-rvg5q\"" Oct 30 22:09:16.920753 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:16.920717 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p"] Oct 30 22:09:16.988887 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:16.988852 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-credential-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:16.988887 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:16.988889 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch8nj\" (UniqueName: \"kubernetes.io/projected/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-kube-api-access-ch8nj\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:16.989129 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:16.989042 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-data\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:16.989129 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:16.989068 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istiod-ca-cert\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:16.989238 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:16.989143 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-workload-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:16.989344 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:16.989315 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-workload-certs\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:16.989406 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:16.989383 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-token\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:16.989470 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:16.989454 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-envoy\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:16.989532 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:16.989517 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-podinfo\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:17.091190 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:17.091088 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-token\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:17.091515 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:17.091488 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-envoy\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:17.091688 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:17.091671 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-podinfo\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:17.091863 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:17.091849 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-credential-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:17.091965 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:17.091951 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8nj\" (UniqueName: \"kubernetes.io/projected/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-kube-api-access-ch8nj\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:17.092080 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:17.092062 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-data\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:17.092209 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:17.092192 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istiod-ca-cert\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:17.092426 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:17.092407 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-workload-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:17.092607 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:17.092588 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-workload-certs\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:17.093073 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:17.093051 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-workload-certs\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:17.093749 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:17.093718 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-data\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:17.097178 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:17.094374 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istiod-ca-cert\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:17.097178 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:17.094622 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-workload-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:17.097178 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:17.092485 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-credential-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:17.097178 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:17.095323 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-envoy\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:17.097178 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:17.097065 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-podinfo\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:17.150282 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:17.150213 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-token\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:17.151498 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:17.151473 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch8nj\" (UniqueName: \"kubernetes.io/projected/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-kube-api-access-ch8nj\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:17.436839 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:17.436745 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:17.869881 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:17.869846 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p"] Oct 30 22:09:17.870925 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:09:17.870895 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa0ffc2_0a81_47ff_83c5_751eff96ac2d.slice/crio-85f4b62f2ca41caaf2bd72db48542272e69b430ff250ab37e51c6cdd3bac8f2d WatchSource:0}: Error finding container 85f4b62f2ca41caaf2bd72db48542272e69b430ff250ab37e51c6cdd3bac8f2d: Status 404 returned error can't find the container with id 85f4b62f2ca41caaf2bd72db48542272e69b430ff250ab37e51c6cdd3bac8f2d Oct 30 22:09:17.873499 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:17.873467 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:09:17.873585 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:17.873544 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:09:17.873585 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:17.873575 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:09:17.901069 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:17.901032 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" event={"ID":"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d","Type":"ContainerStarted","Data":"85f4b62f2ca41caaf2bd72db48542272e69b430ff250ab37e51c6cdd3bac8f2d"} Oct 30 22:09:18.936016 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:18.935968 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" event={"ID":"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d","Type":"ContainerStarted","Data":"69b2526d6aa09976a97f48bd39c2752b245fc511fe5ddbfb1b8126795ebf3f2f"} Oct 30 22:09:19.124274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:19.124098 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" podStartSLOduration=3.124072496 podStartE2EDuration="3.124072496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:19.098492711 +0000 UTC m=+1161.650669012" watchObservedRunningTime="2026-10-30 22:09:19.124072496 +0000 UTC m=+1161.676248798" Oct 30 22:09:19.437347 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:19.437307 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:19.438844 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:19.438816 6006 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:09:19.438997 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:19.438871 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" podUID="5aa0ffc2-0a81-47ff-83c5-751eff96ac2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 22:09:20.437852 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:20.437806 6006 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:09:20.438321 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:20.437882 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" podUID="5aa0ffc2-0a81-47ff-83c5-751eff96ac2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 22:09:21.437367 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:21.437325 6006 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:09:21.437577 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:21.437398 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" podUID="5aa0ffc2-0a81-47ff-83c5-751eff96ac2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 22:09:22.437591 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:22.437551 6006 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:09:22.437969 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:22.437620 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" podUID="5aa0ffc2-0a81-47ff-83c5-751eff96ac2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 22:09:23.437184 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:23.437144 6006 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:09:23.437419 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:23.437211 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" podUID="5aa0ffc2-0a81-47ff-83c5-751eff96ac2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 22:09:24.437619 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:24.437582 6006 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:09:24.438025 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:24.437659 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" podUID="5aa0ffc2-0a81-47ff-83c5-751eff96ac2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 22:09:25.441737 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:25.441707 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:25.442184 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:25.442164 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:25.442938 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:25.442916 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:09:27.912319 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:09:27.912239 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:09:36.681765 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:36.681680 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:09:47.469130 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.469097 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv"] Oct 30 22:09:47.473399 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.473374 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.523415 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.523373 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-fb0b93-openshift-default-dockercfg-2kctd\"" Oct 30 22:09:47.574783 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.574743 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv"] Oct 30 22:09:47.600280 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.600226 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-workload-certs\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.600583 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.600547 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-envoy\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.600722 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.600599 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-data\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.600722 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.600629 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istiod-ca-cert\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.600722 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.600655 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-token\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.600722 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.600696 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-podinfo\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.600948 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.600777 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-workload-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.600948 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.600821 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7xh9\" (UniqueName: \"kubernetes.io/projected/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-kube-api-access-w7xh9\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.600948 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.600939 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-credential-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.701802 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.701754 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7xh9\" (UniqueName: \"kubernetes.io/projected/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-kube-api-access-w7xh9\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.702041 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.702017 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-credential-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.702131 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.702111 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-workload-certs\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.702189 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.702173 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-envoy\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.702242 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.702202 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-data\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.702351 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.702325 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istiod-ca-cert\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.702406 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.702372 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-token\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.702447 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.702416 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-podinfo\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.702539 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.702522 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-workload-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.702601 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.702584 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-workload-certs\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.702657 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.702623 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-data\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.702945 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.702917 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-workload-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.703046 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.702975 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istiod-ca-cert\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.703109 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.703047 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-credential-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.704686 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.704661 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-envoy\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.704901 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.704882 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-podinfo\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.762973 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.762891 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-token\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.763121 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.763081 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7xh9\" (UniqueName: \"kubernetes.io/projected/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-kube-api-access-w7xh9\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:47.787978 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:47.787941 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:48.214189 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:48.214164 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv"] Oct 30 22:09:48.215725 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:09:48.215693 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9cf1a6_b28c_4ec7_8e72_4205ae0982c6.slice/crio-c9cc07407ee84b298be6e208f0ba620117e5883165860c1140cac089b0ed6333 WatchSource:0}: Error finding container c9cc07407ee84b298be6e208f0ba620117e5883165860c1140cac089b0ed6333: Status 404 returned error can't find the container with id c9cc07407ee84b298be6e208f0ba620117e5883165860c1140cac089b0ed6333 Oct 30 22:09:48.218336 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:48.218301 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:09:48.218427 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:48.218371 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:09:48.218427 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:48.218401 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:09:49.137628 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:49.137586 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" event={"ID":"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6","Type":"ContainerStarted","Data":"1c8c6a047107f5731955e18abb5ab34dee947652edcb3fa9e97321ac5105ed23"} Oct 30 22:09:49.138082 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:49.137648 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" event={"ID":"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6","Type":"ContainerStarted","Data":"c9cc07407ee84b298be6e208f0ba620117e5883165860c1140cac089b0ed6333"} Oct 30 22:09:49.788183 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:49.788140 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:49.790332 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:49.790303 6006 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:09:49.790459 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:49.790361 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" podUID="4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 22:09:50.788863 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:50.788824 6006 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:09:50.789296 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:50.788879 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" podUID="4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 22:09:51.788745 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:51.788706 6006 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:09:51.788961 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:51.788810 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" podUID="4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 22:09:52.788521 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:52.788483 6006 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:09:52.788840 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:52.788546 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" podUID="4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 22:09:53.788877 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:53.788832 6006 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:09:53.789415 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:53.788905 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" podUID="4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 22:09:54.789036 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:54.788988 6006 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:09:54.789579 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:54.789061 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" podUID="4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 22:09:55.793440 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:55.793407 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:55.793948 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:55.793925 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:55.794775 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:55.794755 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:09:55.956784 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:55.956716 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" podStartSLOduration=8.956699596 podStartE2EDuration="8.956699596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:49.313053122 +0000 UTC m=+1191.865229423" watchObservedRunningTime="2026-10-30 22:09:55.956699596 +0000 UTC m=+1198.508875899" Oct 30 22:09:57.756321 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:57.756293 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 22:09:57.757824 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:57.757790 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 22:09:57.911620 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:09:57.911572 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:09:58.051155 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:58.051117 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886"] Oct 30 22:09:58.056399 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:58.056377 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:09:58.113194 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:58.113157 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886"] Oct 30 22:09:58.138081 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:58.138033 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a00d9b\"/\"distinct-w1-a00d9b-kserve-self-signed-certs\"" Oct 30 22:09:58.145372 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:58.145329 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-home\") pod \"distinct-w1-a00d9b-kserve-85f69cb446-ln886\" (UID: \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:09:58.145575 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:58.145397 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fjhm\" (UniqueName: \"kubernetes.io/projected/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-kube-api-access-8fjhm\") pod \"distinct-w1-a00d9b-kserve-85f69cb446-ln886\" (UID: \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:09:58.145575 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:58.145520 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-tls-certs\") pod \"distinct-w1-a00d9b-kserve-85f69cb446-ln886\" (UID: \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:09:58.145664 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:58.145583 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-model-cache\") pod \"distinct-w1-a00d9b-kserve-85f69cb446-ln886\" (UID: \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:09:58.145702 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:58.145679 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-dshm\") pod \"distinct-w1-a00d9b-kserve-85f69cb446-ln886\" (UID: \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:09:58.145743 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:58.145713 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-tmp-dir\") pod \"distinct-w1-a00d9b-kserve-85f69cb446-ln886\" (UID: \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:09:58.246919 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:58.246881 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-home\") pod \"distinct-w1-a00d9b-kserve-85f69cb446-ln886\" (UID: \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:09:58.247165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:58.246934 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fjhm\" (UniqueName: \"kubernetes.io/projected/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-kube-api-access-8fjhm\") pod \"distinct-w1-a00d9b-kserve-85f69cb446-ln886\" (UID: \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:09:58.247165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:58.246974 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-tls-certs\") pod \"distinct-w1-a00d9b-kserve-85f69cb446-ln886\" (UID: \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:09:58.247165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:58.247004 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-model-cache\") pod \"distinct-w1-a00d9b-kserve-85f69cb446-ln886\" (UID: \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:09:58.247165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:58.247034 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-dshm\") pod \"distinct-w1-a00d9b-kserve-85f69cb446-ln886\" (UID: \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:09:58.247165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:58.247054 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-tmp-dir\") pod \"distinct-w1-a00d9b-kserve-85f69cb446-ln886\" (UID: \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:09:58.247570 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:58.247549 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-tmp-dir\") pod \"distinct-w1-a00d9b-kserve-85f69cb446-ln886\" (UID: \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:09:58.247769 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:58.247752 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-home\") pod \"distinct-w1-a00d9b-kserve-85f69cb446-ln886\" (UID: \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:09:58.248689 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:58.248665 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-model-cache\") pod \"distinct-w1-a00d9b-kserve-85f69cb446-ln886\" (UID: \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:09:58.250831 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:58.250807 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-dshm\") pod \"distinct-w1-a00d9b-kserve-85f69cb446-ln886\" (UID: \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:09:58.251265 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:58.251226 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-tls-certs\") pod \"distinct-w1-a00d9b-kserve-85f69cb446-ln886\" (UID: \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:09:58.306519 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:58.306432 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fjhm\" (UniqueName: \"kubernetes.io/projected/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-kube-api-access-8fjhm\") pod \"distinct-w1-a00d9b-kserve-85f69cb446-ln886\" (UID: \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\") " pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:09:58.369911 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:58.369872 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:09:58.784908 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:58.784878 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886"] Oct 30 22:09:58.786058 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:09:58.786031 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2715f84_b3fa_4d3d_80f9_18b5c8a7359a.slice/crio-91b6bb3f7ba7ba763fdbeb4f47bc8166ecb52a589cbe2f5623d32ad979f65233 WatchSource:0}: Error finding container 91b6bb3f7ba7ba763fdbeb4f47bc8166ecb52a589cbe2f5623d32ad979f65233: Status 404 returned error can't find the container with id 91b6bb3f7ba7ba763fdbeb4f47bc8166ecb52a589cbe2f5623d32ad979f65233 Oct 30 22:09:59.218516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:59.218475 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" event={"ID":"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a","Type":"ContainerStarted","Data":"da7156d8339ec9488e38599efafa17081e9925750125c778e6d950a7ded1e255"} Oct 30 22:09:59.219337 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:59.218527 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" event={"ID":"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a","Type":"ContainerStarted","Data":"91b6bb3f7ba7ba763fdbeb4f47bc8166ecb52a589cbe2f5623d32ad979f65233"} Oct 30 22:09:59.335627 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:59.335474 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" podStartSLOduration=1.335453458 podStartE2EDuration="1.335453458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:59.333572305 +0000 UTC m=+1201.885748606" watchObservedRunningTime="2026-10-30 22:09:59.335453458 +0000 UTC m=+1201.887629760" Oct 30 22:09:59.370976 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:09:59.370942 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:10:08.370422 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:08.370380 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:10:08.383101 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:08.383062 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:10:09.300025 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:09.299992 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:10:11.220271 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:11.220169 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:10:20.127216 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.127180 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp"] Oct 30 22:10:20.131298 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.131281 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:10:20.194000 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.193951 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp"] Oct 30 22:10:20.231230 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.231196 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-65e59f\"/\"openshift-service-ca.crt\"" Oct 30 22:10:20.231429 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.231239 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-65e59f\"/\"kube-root-ca.crt\"" Oct 30 22:10:20.231429 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.231350 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-65e59f\"/\"distinct-w4-65e59f-kserve-self-signed-certs\"" Oct 30 22:10:20.231429 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.231364 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-65e59f\"/\"default-dockercfg-zpz5r\"" Oct 30 22:10:20.262732 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.262692 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/636d5b64-a287-4a2b-bee0-db3b0184b0f1-tls-certs\") pod \"distinct-w4-65e59f-kserve-58947cc74f-r2tsp\" (UID: \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:10:20.262932 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.262749 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-home\") pod \"distinct-w4-65e59f-kserve-58947cc74f-r2tsp\" (UID: \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:10:20.263003 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.262930 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-dshm\") pod \"distinct-w4-65e59f-kserve-58947cc74f-r2tsp\" (UID: \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:10:20.263003 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.262986 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-model-cache\") pod \"distinct-w4-65e59f-kserve-58947cc74f-r2tsp\" (UID: \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:10:20.263103 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.263050 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lpgw\" (UniqueName: \"kubernetes.io/projected/636d5b64-a287-4a2b-bee0-db3b0184b0f1-kube-api-access-7lpgw\") pod \"distinct-w4-65e59f-kserve-58947cc74f-r2tsp\" (UID: \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:10:20.263103 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.263094 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-tmp-dir\") pod \"distinct-w4-65e59f-kserve-58947cc74f-r2tsp\" (UID: \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:10:20.364499 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.364460 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lpgw\" (UniqueName: \"kubernetes.io/projected/636d5b64-a287-4a2b-bee0-db3b0184b0f1-kube-api-access-7lpgw\") pod \"distinct-w4-65e59f-kserve-58947cc74f-r2tsp\" (UID: \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:10:20.364704 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.364520 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-tmp-dir\") pod \"distinct-w4-65e59f-kserve-58947cc74f-r2tsp\" (UID: \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:10:20.364704 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.364664 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/636d5b64-a287-4a2b-bee0-db3b0184b0f1-tls-certs\") pod \"distinct-w4-65e59f-kserve-58947cc74f-r2tsp\" (UID: \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:10:20.364818 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.364716 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-home\") pod \"distinct-w4-65e59f-kserve-58947cc74f-r2tsp\" (UID: \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:10:20.364818 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.364792 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-dshm\") pod \"distinct-w4-65e59f-kserve-58947cc74f-r2tsp\" (UID: \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:10:20.364891 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.364832 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-model-cache\") pod \"distinct-w4-65e59f-kserve-58947cc74f-r2tsp\" (UID: \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:10:20.365057 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.365026 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-tmp-dir\") pod \"distinct-w4-65e59f-kserve-58947cc74f-r2tsp\" (UID: \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:10:20.365191 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.365165 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-home\") pod \"distinct-w4-65e59f-kserve-58947cc74f-r2tsp\" (UID: \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:10:20.365433 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.365237 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-model-cache\") pod \"distinct-w4-65e59f-kserve-58947cc74f-r2tsp\" (UID: \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:10:20.367446 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.367425 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/636d5b64-a287-4a2b-bee0-db3b0184b0f1-tls-certs\") pod \"distinct-w4-65e59f-kserve-58947cc74f-r2tsp\" (UID: \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:10:20.367553 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.367431 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-dshm\") pod \"distinct-w4-65e59f-kserve-58947cc74f-r2tsp\" (UID: \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:10:20.423734 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.423652 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lpgw\" (UniqueName: \"kubernetes.io/projected/636d5b64-a287-4a2b-bee0-db3b0184b0f1-kube-api-access-7lpgw\") pod \"distinct-w4-65e59f-kserve-58947cc74f-r2tsp\" (UID: \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\") " pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:10:20.445225 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.445189 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:10:20.885353 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:20.885319 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp"] Oct 30 22:10:20.886505 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:10:20.886473 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod636d5b64_a287_4a2b_bee0_db3b0184b0f1.slice/crio-7ff4cfe3a188d4e125e4a39436d34a8ac21ec7701dcc71eae7c43ed69c4be227 WatchSource:0}: Error finding container 7ff4cfe3a188d4e125e4a39436d34a8ac21ec7701dcc71eae7c43ed69c4be227: Status 404 returned error can't find the container with id 7ff4cfe3a188d4e125e4a39436d34a8ac21ec7701dcc71eae7c43ed69c4be227 Oct 30 22:10:21.401803 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:21.401762 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" event={"ID":"636d5b64-a287-4a2b-bee0-db3b0184b0f1","Type":"ContainerStarted","Data":"1e4b39dcd309429e18613b9593aaa5b1372fca9c9cf0444663d42d3a7c32e203"} Oct 30 22:10:21.402346 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:21.401813 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" event={"ID":"636d5b64-a287-4a2b-bee0-db3b0184b0f1","Type":"ContainerStarted","Data":"7ff4cfe3a188d4e125e4a39436d34a8ac21ec7701dcc71eae7c43ed69c4be227"} Oct 30 22:10:21.445524 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:21.445486 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:10:21.520214 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:21.520154 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" podStartSLOduration=1.52013758 podStartE2EDuration="1.52013758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:21.516837603 +0000 UTC m=+1224.069013916" watchObservedRunningTime="2026-10-30 22:10:21.52013758 +0000 UTC m=+1224.072313876" Oct 30 22:10:27.913398 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:10:27.913357 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:10:30.445441 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:30.445392 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:10:30.461603 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:30.461565 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:10:30.473178 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:30.473145 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:10:31.720273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:31.720214 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9"] Oct 30 22:10:31.724616 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:31.724595 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:31.767122 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:31.767088 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc"] Oct 30 22:10:31.771100 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:31.771080 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:31.823832 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:31.823794 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9"] Oct 30 22:10:31.872326 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:31.872287 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc"] Oct 30 22:10:31.911875 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:31.911832 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-workload-certs\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:31.912073 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:31.911946 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-workload-certs\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:31.912073 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:31.911998 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-envoy\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:31.912160 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:31.912101 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f33ebace-e652-4f07-bbce-9c334faa3948-istiod-ca-cert\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:31.912199 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:31.912165 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-data\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:31.912199 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:31.912193 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f33ebace-e652-4f07-bbce-9c334faa3948-istio-podinfo\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:31.912293 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:31.912223 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-token\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:31.912328 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:31.912308 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-istio-data\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:31.912364 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:31.912350 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-workload-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:31.912410 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:31.912380 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-workload-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:31.912447 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:31.912413 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct25w\" (UniqueName: \"kubernetes.io/projected/f33ebace-e652-4f07-bbce-9c334faa3948-kube-api-access-ct25w\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:31.912480 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:31.912448 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f33ebace-e652-4f07-bbce-9c334faa3948-istio-token\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:31.912513 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:31.912482 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-podinfo\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:31.912555 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:31.912509 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnzqg\" (UniqueName: \"kubernetes.io/projected/46a061d3-8105-4260-9bf1-4fc3a5a09343-kube-api-access-tnzqg\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:31.912607 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:31.912553 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-istio-envoy\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:31.912657 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:31.912609 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-credential-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:31.912657 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:31.912633 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-credential-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:31.912754 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:31.912689 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46a061d3-8105-4260-9bf1-4fc3a5a09343-istiod-ca-cert\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:32.014434 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.014329 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-workload-certs\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:32.014434 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.014375 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-workload-certs\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:32.014434 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.014413 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-envoy\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:32.014726 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.014451 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f33ebace-e652-4f07-bbce-9c334faa3948-istiod-ca-cert\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:32.014726 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.014484 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-data\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:32.014726 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.014511 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f33ebace-e652-4f07-bbce-9c334faa3948-istio-podinfo\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:32.014726 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.014539 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-token\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:32.014726 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.014589 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-istio-data\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:32.014726 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.014625 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-workload-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:32.014726 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.014652 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-workload-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:32.014726 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.014686 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct25w\" (UniqueName: \"kubernetes.io/projected/f33ebace-e652-4f07-bbce-9c334faa3948-kube-api-access-ct25w\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:32.015149 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.014728 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f33ebace-e652-4f07-bbce-9c334faa3948-istio-token\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:32.015149 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.014765 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-podinfo\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:32.015149 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.014795 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnzqg\" (UniqueName: \"kubernetes.io/projected/46a061d3-8105-4260-9bf1-4fc3a5a09343-kube-api-access-tnzqg\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:32.015149 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.014846 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-istio-envoy\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:32.015149 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.014905 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-credential-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:32.015149 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.014921 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-workload-certs\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:32.015149 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.014931 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-credential-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:32.015149 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.015005 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46a061d3-8105-4260-9bf1-4fc3a5a09343-istiod-ca-cert\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:32.015597 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.015237 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-data\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:32.015597 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.015311 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f33ebace-e652-4f07-bbce-9c334faa3948-istiod-ca-cert\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:32.015597 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.015547 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-credential-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:32.015763 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.015739 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46a061d3-8105-4260-9bf1-4fc3a5a09343-istiod-ca-cert\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:32.016085 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.016011 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-istio-data\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:32.019110 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.016240 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-workload-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:32.019346 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.019135 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-workload-certs\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:32.024272 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.019449 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-istio-envoy\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:32.024272 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.019610 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-workload-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:32.024272 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.019725 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f33ebace-e652-4f07-bbce-9c334faa3948-istio-podinfo\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:32.024272 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.019786 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-envoy\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:32.024272 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.019898 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-credential-socket\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:32.024272 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.020456 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-podinfo\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:32.074554 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.074516 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-token\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:32.074739 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.074631 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct25w\" (UniqueName: \"kubernetes.io/projected/f33ebace-e652-4f07-bbce-9c334faa3948-kube-api-access-ct25w\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:32.074810 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.074777 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f33ebace-e652-4f07-bbce-9c334faa3948-istio-token\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:32.075017 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.074994 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnzqg\" (UniqueName: \"kubernetes.io/projected/46a061d3-8105-4260-9bf1-4fc3a5a09343-kube-api-access-tnzqg\") pod \"e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:32.084466 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.084440 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:32.197607 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.197546 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6"] Oct 30 22:10:32.204877 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.204843 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.257656 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.257623 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6"] Oct 30 22:10:32.319063 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.319019 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-podinfo\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.319327 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.319072 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-token\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.319385 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.319326 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-workload-certs\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.319385 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.319371 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb25e8f-893d-486f-aed7-922d9804d9c9-istiod-ca-cert\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.319460 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.319400 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j48xn\" (UniqueName: \"kubernetes.io/projected/2cb25e8f-893d-486f-aed7-922d9804d9c9-kube-api-access-j48xn\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.319508 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.319485 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-credential-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.319670 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.319649 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-envoy\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.319735 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.319715 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-workload-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.319784 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.319770 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-data\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.340115 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.340054 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:32.421482 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.420910 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-workload-certs\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.421482 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.420967 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb25e8f-893d-486f-aed7-922d9804d9c9-istiod-ca-cert\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.421482 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.421000 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j48xn\" (UniqueName: \"kubernetes.io/projected/2cb25e8f-893d-486f-aed7-922d9804d9c9-kube-api-access-j48xn\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.421482 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.421082 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-credential-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.421482 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.421140 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-envoy\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.421482 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.421177 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-workload-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.421482 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.421236 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-data\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.421482 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.421319 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-podinfo\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.421482 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.421353 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-token\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.423073 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.422571 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-workload-certs\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.423073 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.422661 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-workload-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.423073 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.422828 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-data\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.423363 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.423131 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-credential-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.423363 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.423276 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb25e8f-893d-486f-aed7-922d9804d9c9-istiod-ca-cert\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.426039 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.425994 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-podinfo\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.426654 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.426614 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-envoy\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.482907 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.482868 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-token\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.483067 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.482917 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j48xn\" (UniqueName: \"kubernetes.io/projected/2cb25e8f-893d-486f-aed7-922d9804d9c9-kube-api-access-j48xn\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.517855 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.517821 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc"] Oct 30 22:10:32.519428 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:10:32.519394 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a061d3_8105_4260_9bf1_4fc3a5a09343.slice/crio-0161029169b6a4955eae40d5fe6f5d7f9e6e7639d242ef776697863b83c0fa3d WatchSource:0}: Error finding container 0161029169b6a4955eae40d5fe6f5d7f9e6e7639d242ef776697863b83c0fa3d: Status 404 returned error can't find the container with id 0161029169b6a4955eae40d5fe6f5d7f9e6e7639d242ef776697863b83c0fa3d Oct 30 22:10:32.522013 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.521965 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:10:32.522108 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.522066 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:10:32.522149 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.522107 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:10:32.525424 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.525400 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:32.724052 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:10:32.724014 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33ebace_e652_4f07_bbce_9c334faa3948.slice/crio-8957b8c9d255a0621c22c8de0a48a021d4be9542753dcbea392ec1bc9643a59b WatchSource:0}: Error finding container 8957b8c9d255a0621c22c8de0a48a021d4be9542753dcbea392ec1bc9643a59b: Status 404 returned error can't find the container with id 8957b8c9d255a0621c22c8de0a48a021d4be9542753dcbea392ec1bc9643a59b Oct 30 22:10:32.724903 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.724877 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9"] Oct 30 22:10:32.727074 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.727036 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:10:32.727184 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.727119 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:10:32.727184 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.727166 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:10:32.911780 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:10:32.911750 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb25e8f_893d_486f_aed7_922d9804d9c9.slice/crio-c48dc50022b0fb3b46780b81a6b625bd26e3b7e4d1eba9bf964e2eb05baa616a WatchSource:0}: Error finding container c48dc50022b0fb3b46780b81a6b625bd26e3b7e4d1eba9bf964e2eb05baa616a: Status 404 returned error can't find the container with id c48dc50022b0fb3b46780b81a6b625bd26e3b7e4d1eba9bf964e2eb05baa616a Oct 30 22:10:32.912649 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.912616 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6"] Oct 30 22:10:32.914710 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.914669 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:10:32.914791 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.914757 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:10:32.914840 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:32.914823 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:10:33.482498 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:33.482460 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" event={"ID":"f33ebace-e652-4f07-bbce-9c334faa3948","Type":"ContainerStarted","Data":"a7f28a559a912c74ad8c03267b0ae6427e548a2e7c3a57364b48b76ce5dc6f12"} Oct 30 22:10:33.482731 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:33.482526 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" event={"ID":"f33ebace-e652-4f07-bbce-9c334faa3948","Type":"ContainerStarted","Data":"8957b8c9d255a0621c22c8de0a48a021d4be9542753dcbea392ec1bc9643a59b"} Oct 30 22:10:33.484175 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:33.484145 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" event={"ID":"2cb25e8f-893d-486f-aed7-922d9804d9c9","Type":"ContainerStarted","Data":"b9e4c73a51ed3bae59bcc8585870682149154c76278a5a0ff30624a82693e525"} Oct 30 22:10:33.484328 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:33.484196 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" event={"ID":"2cb25e8f-893d-486f-aed7-922d9804d9c9","Type":"ContainerStarted","Data":"c48dc50022b0fb3b46780b81a6b625bd26e3b7e4d1eba9bf964e2eb05baa616a"} Oct 30 22:10:33.485807 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:33.485773 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" event={"ID":"46a061d3-8105-4260-9bf1-4fc3a5a09343","Type":"ContainerStarted","Data":"fbc2fbcbea5902da8911c902b7dc076c87011d543a001f4730bd3588bacc4790"} Oct 30 22:10:33.485911 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:33.485827 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" event={"ID":"46a061d3-8105-4260-9bf1-4fc3a5a09343","Type":"ContainerStarted","Data":"0161029169b6a4955eae40d5fe6f5d7f9e6e7639d242ef776697863b83c0fa3d"} Oct 30 22:10:33.535280 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:33.533752 6006 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:10:33.535280 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:33.533820 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" podUID="2cb25e8f-893d-486f-aed7-922d9804d9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 22:10:33.535280 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:33.535059 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:33.613292 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:33.604762 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" podStartSLOduration=2.604739048 podStartE2EDuration="2.604739048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:33.604584714 +0000 UTC m=+1236.156761016" watchObservedRunningTime="2026-10-30 22:10:33.604739048 +0000 UTC m=+1236.156915352" Oct 30 22:10:33.735334 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:33.734204 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" podStartSLOduration=2.734181059 podStartE2EDuration="2.734181059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:33.72078809 +0000 UTC m=+1236.272964393" watchObservedRunningTime="2026-10-30 22:10:33.734181059 +0000 UTC m=+1236.286357367" Oct 30 22:10:33.846329 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:33.845750 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" podStartSLOduration=1.845726966 podStartE2EDuration="1.845726966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:33.836288202 +0000 UTC m=+1236.388464503" watchObservedRunningTime="2026-10-30 22:10:33.845726966 +0000 UTC m=+1236.397903270" Oct 30 22:10:34.092273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:34.091539 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:34.092273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:34.091605 6006 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:10:34.092273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:34.091679 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" podUID="46a061d3-8105-4260-9bf1-4fc3a5a09343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 22:10:34.344336 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:34.343633 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:34.344336 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:34.343789 6006 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:10:34.344336 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:34.343834 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" podUID="f33ebace-e652-4f07-bbce-9c334faa3948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 22:10:34.541275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:34.540782 6006 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:10:34.541275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:34.540859 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" podUID="2cb25e8f-893d-486f-aed7-922d9804d9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 22:10:35.087289 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:35.086724 6006 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:10:35.087289 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:35.086796 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" podUID="46a061d3-8105-4260-9bf1-4fc3a5a09343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 22:10:35.342330 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:35.341673 6006 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:10:35.342330 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:35.341738 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" podUID="f33ebace-e652-4f07-bbce-9c334faa3948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 22:10:35.530326 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:35.529407 6006 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:10:35.530326 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:35.529469 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" podUID="2cb25e8f-893d-486f-aed7-922d9804d9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 22:10:36.086273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:36.085706 6006 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:10:36.086273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:36.085773 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" podUID="46a061d3-8105-4260-9bf1-4fc3a5a09343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 22:10:36.343333 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:36.342843 6006 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:10:36.343333 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:36.342953 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" podUID="f33ebace-e652-4f07-bbce-9c334faa3948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 22:10:36.551273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:36.550529 6006 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:10:36.551273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:36.550598 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" podUID="2cb25e8f-893d-486f-aed7-922d9804d9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 22:10:37.086283 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:37.085665 6006 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:10:37.086283 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:37.085727 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" podUID="46a061d3-8105-4260-9bf1-4fc3a5a09343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 22:10:37.347217 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:37.346673 6006 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:10:37.347217 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:37.346741 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" podUID="f33ebace-e652-4f07-bbce-9c334faa3948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 22:10:37.527277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:37.526795 6006 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:10:37.527277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:37.526873 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" podUID="2cb25e8f-893d-486f-aed7-922d9804d9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 22:10:38.086698 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:38.086654 6006 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:10:38.086975 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:38.086949 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" podUID="46a061d3-8105-4260-9bf1-4fc3a5a09343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 22:10:38.434785 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:38.434690 6006 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:10:38.474860 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:38.474813 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" podUID="f33ebace-e652-4f07-bbce-9c334faa3948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 22:10:38.475106 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:38.442877 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 22:10:38.619276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:38.611707 6006 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:10:38.619276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:38.611793 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" podUID="2cb25e8f-893d-486f-aed7-922d9804d9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 22:10:39.086271 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:39.085656 6006 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:10:39.086271 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:39.085728 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" podUID="46a061d3-8105-4260-9bf1-4fc3a5a09343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 22:10:39.343427 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:39.343157 6006 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:10:39.343427 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:39.343234 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" podUID="f33ebace-e652-4f07-bbce-9c334faa3948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 22:10:39.453417 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:39.453359 6006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:39.470422 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:39.470362 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:39.471071 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:39.471033 6006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:39.471816 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:39.471705 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 22:10:39.534692 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:39.534648 6006 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:10:39.534999 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:39.534972 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" podUID="2cb25e8f-893d-486f-aed7-922d9804d9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 22:10:39.762338 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:39.753709 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:40.012877 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:40.012157 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 22:10:40.086615 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:40.086578 6006 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:10:40.086904 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:40.086871 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" podUID="46a061d3-8105-4260-9bf1-4fc3a5a09343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 22:10:40.341285 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:40.340787 6006 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:10:40.341285 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:40.340867 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" podUID="f33ebace-e652-4f07-bbce-9c334faa3948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 22:10:40.370271 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:40.360386 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:40.475515 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:40.475413 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:40.477401 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:40.476279 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:10:40.477779 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:40.477751 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:40.479066 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:40.478285 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:10:40.531442 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:40.531402 6006 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:10:40.531720 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:40.531693 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" podUID="2cb25e8f-893d-486f-aed7-922d9804d9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 22:10:40.673511 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:40.673401 6006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded" Oct 30 22:10:40.773393 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:40.757299 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:40.792288 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:40.773756 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:10:40.930331 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:40.923042 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:10:40.930331 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:40.923593 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:10:41.105507 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:41.105455 6006 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:10:41.105817 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:41.105781 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" podUID="46a061d3-8105-4260-9bf1-4fc3a5a09343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 22:10:41.122175 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:41.122143 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:10:41.384418 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:41.384384 6006 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:10:41.384712 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:41.384686 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" podUID="f33ebace-e652-4f07-bbce-9c334faa3948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 22:10:41.589205 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:41.588137 6006 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:10:41.593533 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:41.592672 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" podUID="2cb25e8f-893d-486f-aed7-922d9804d9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 22:10:42.092562 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:42.092524 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:42.092982 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:42.092960 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:42.094777 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:42.094755 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:10:42.347951 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:42.347869 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:42.349748 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:42.349724 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:42.351198 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:42.351173 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:10:42.739994 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:42.739917 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:42.740451 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:42.740404 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:42.741236 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:10:42.741215 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:10:57.913650 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:10:57.913603 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:11:00.956627 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:00.956589 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942"] Oct 30 22:11:00.962075 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:00.962051 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:00.981404 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:00.981368 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-token\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:00.981558 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:00.981430 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-workload-certs\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:00.981600 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:00.981554 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-workload-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:00.981642 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:00.981593 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-data\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:00.981676 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:00.981647 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cm9b\" (UniqueName: \"kubernetes.io/projected/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-kube-api-access-7cm9b\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:00.981676 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:00.981671 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-podinfo\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:00.981739 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:00.981709 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-envoy\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:00.981779 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:00.981742 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-credential-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:00.981779 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:00.981767 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istiod-ca-cert\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:00.998929 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:00.998894 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw"] Oct 30 22:11:01.003607 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.003582 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.054979 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.054942 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942"] Oct 30 22:11:01.082711 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.082670 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-workload-certs\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:01.082892 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.082726 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-data\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.082892 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.082789 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4a885f9-5fd5-4782-978b-ceb9fb740610-istiod-ca-cert\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.082892 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.082843 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-workload-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:01.082892 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.082861 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n9m7\" (UniqueName: \"kubernetes.io/projected/d4a885f9-5fd5-4782-978b-ceb9fb740610-kube-api-access-2n9m7\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.082892 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.082883 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-workload-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.083143 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.082912 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-data\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:01.083143 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.082928 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-token\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.083143 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.082948 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cm9b\" (UniqueName: \"kubernetes.io/projected/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-kube-api-access-7cm9b\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:01.083143 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.082970 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-podinfo\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:01.083143 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.082997 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-envoy\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:01.083143 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.083036 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-credential-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:01.083143 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.083060 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istiod-ca-cert\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:01.083143 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.083093 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-podinfo\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.083143 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.083128 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-envoy\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.083476 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.083194 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-credential-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.083476 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.083299 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-token\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:01.083476 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.083348 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-workload-certs\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.083754 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.083735 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-workload-certs\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:01.083957 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.083940 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-workload-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:01.084165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.084137 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-data\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:01.084438 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.084400 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-credential-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:01.084831 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.084780 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istiod-ca-cert\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:01.088600 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.088571 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-envoy\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:01.088771 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.088753 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-podinfo\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:01.144734 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.144698 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-token\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:01.151425 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.151388 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw"] Oct 30 22:11:01.152393 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.152360 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cm9b\" (UniqueName: \"kubernetes.io/projected/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-kube-api-access-7cm9b\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:01.184267 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.184215 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-podinfo\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.184460 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.184296 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-envoy\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.184460 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.184366 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-credential-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.184634 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.184590 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-workload-certs\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.184801 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.184733 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-data\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.184879 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.184813 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4a885f9-5fd5-4782-978b-ceb9fb740610-istiod-ca-cert\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.184940 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.184871 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-credential-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.184940 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.184889 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n9m7\" (UniqueName: \"kubernetes.io/projected/d4a885f9-5fd5-4782-978b-ceb9fb740610-kube-api-access-2n9m7\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.184940 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.184913 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-workload-certs\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.184940 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.184934 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-workload-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.185157 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.184979 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-token\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.185157 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.185071 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-data\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.185470 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.185446 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-workload-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.185579 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.185560 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4a885f9-5fd5-4782-978b-ceb9fb740610-istiod-ca-cert\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.186863 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.186841 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-envoy\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.187478 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.187457 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-podinfo\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.244129 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.244039 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-token\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.244288 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.244164 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n9m7\" (UniqueName: \"kubernetes.io/projected/d4a885f9-5fd5-4782-978b-ceb9fb740610-kube-api-access-2n9m7\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.276296 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.276260 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:01.318433 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.318386 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:01.655366 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.655326 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf"] Oct 30 22:11:01.660544 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.660504 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.691361 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.691325 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8033a54b-e69c-49d8-ad42-315c1486179a-istio-podinfo\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.691361 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.691363 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-istio-envoy\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.691607 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.691475 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-credential-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.691607 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.691589 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-istio-data\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.691698 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.691615 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfbgn\" (UniqueName: \"kubernetes.io/projected/8033a54b-e69c-49d8-ad42-315c1486179a-kube-api-access-wfbgn\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.691735 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.691724 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8033a54b-e69c-49d8-ad42-315c1486179a-istiod-ca-cert\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.691782 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.691768 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-workload-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.691819 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.691808 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8033a54b-e69c-49d8-ad42-315c1486179a-istio-token\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.691884 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.691858 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-workload-certs\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.763081 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:11:01.763033 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdbec1e1_b705_464b_88bd_6b6ddbfb3513.slice/crio-a2d137330f1dba90db30846f0fb058d6b3f6a96abd5220e135cee4d309ccc2be WatchSource:0}: Error finding container a2d137330f1dba90db30846f0fb058d6b3f6a96abd5220e135cee4d309ccc2be: Status 404 returned error can't find the container with id a2d137330f1dba90db30846f0fb058d6b3f6a96abd5220e135cee4d309ccc2be Oct 30 22:11:01.763461 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.763432 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942"] Oct 30 22:11:01.765999 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.765937 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:11:01.766135 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.766030 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:11:01.766135 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.766077 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:11:01.767287 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.767260 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf"] Oct 30 22:11:01.792689 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.792655 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8033a54b-e69c-49d8-ad42-315c1486179a-istio-podinfo\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.792857 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.792697 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-istio-envoy\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.792857 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.792755 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-credential-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.792857 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.792832 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-istio-data\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.793019 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.792860 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfbgn\" (UniqueName: \"kubernetes.io/projected/8033a54b-e69c-49d8-ad42-315c1486179a-kube-api-access-wfbgn\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.793019 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.792989 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8033a54b-e69c-49d8-ad42-315c1486179a-istiod-ca-cert\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.793127 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.793027 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-workload-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.793127 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.793087 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8033a54b-e69c-49d8-ad42-315c1486179a-istio-token\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.793127 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.793121 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-workload-certs\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.793330 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.793293 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-credential-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.793542 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.793489 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-workload-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.793671 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.793568 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-workload-certs\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.793774 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.793746 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-istio-data\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.794020 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.793995 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8033a54b-e69c-49d8-ad42-315c1486179a-istiod-ca-cert\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.795231 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.795208 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-istio-envoy\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.795509 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.795491 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8033a54b-e69c-49d8-ad42-315c1486179a-istio-podinfo\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.828829 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:11:01.828789 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4a885f9_5fd5_4782_978b_ceb9fb740610.slice/crio-58ecda98cbb10e4487ed84e0b18b47dd9d630436ab8111d73ac6b8214cbb2509 WatchSource:0}: Error finding container 58ecda98cbb10e4487ed84e0b18b47dd9d630436ab8111d73ac6b8214cbb2509: Status 404 returned error can't find the container with id 58ecda98cbb10e4487ed84e0b18b47dd9d630436ab8111d73ac6b8214cbb2509 Oct 30 22:11:01.829664 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.829635 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw"] Oct 30 22:11:01.831068 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.831026 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:11:01.831186 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.831116 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:11:01.831186 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.831161 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:11:01.872560 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.872528 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8033a54b-e69c-49d8-ad42-315c1486179a-istio-token\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.872735 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.872587 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfbgn\" (UniqueName: \"kubernetes.io/projected/8033a54b-e69c-49d8-ad42-315c1486179a-kube-api-access-wfbgn\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:01.976007 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:01.975770 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:02.191566 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:02.191466 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" event={"ID":"d4a885f9-5fd5-4782-978b-ceb9fb740610","Type":"ContainerStarted","Data":"0c659a0ec05b303725b1b8ba27a92dfc351c1d382bc21a171eb29baf8eca6e2f"} Oct 30 22:11:02.191566 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:02.191538 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" event={"ID":"d4a885f9-5fd5-4782-978b-ceb9fb740610","Type":"ContainerStarted","Data":"58ecda98cbb10e4487ed84e0b18b47dd9d630436ab8111d73ac6b8214cbb2509"} Oct 30 22:11:02.193558 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:02.193526 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" event={"ID":"bdbec1e1-b705-464b-88bd-6b6ddbfb3513","Type":"ContainerStarted","Data":"94858957c1fa3efe4e033dd3c7eb1043af3f042f9e6a331f60c29b01339c6d9b"} Oct 30 22:11:02.193678 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:02.193591 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" event={"ID":"bdbec1e1-b705-464b-88bd-6b6ddbfb3513","Type":"ContainerStarted","Data":"a2d137330f1dba90db30846f0fb058d6b3f6a96abd5220e135cee4d309ccc2be"} Oct 30 22:11:02.276870 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:02.276837 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:02.277973 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:02.277945 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:11:02.278076 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:02.278006 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" podUID="bdbec1e1-b705-464b-88bd-6b6ddbfb3513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 22:11:02.310812 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:02.310748 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" podStartSLOduration=2.310729395 podStartE2EDuration="2.310729395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:02.308992942 +0000 UTC m=+1264.861169245" watchObservedRunningTime="2026-10-30 22:11:02.310729395 +0000 UTC m=+1264.862905696" Oct 30 22:11:02.319368 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:02.319328 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:02.321579 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:02.321552 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 22:11:02.321732 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:02.321601 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" podUID="d4a885f9-5fd5-4782-978b-ceb9fb740610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 22:11:02.380790 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:02.380761 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf"] Oct 30 22:11:02.382354 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:11:02.382321 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8033a54b_e69c_49d8_ad42_315c1486179a.slice/crio-d721cc7cfd230de0adbba8bfea2c55a724815412002d2cafb83109af366873ee WatchSource:0}: Error finding container d721cc7cfd230de0adbba8bfea2c55a724815412002d2cafb83109af366873ee: Status 404 returned error can't find the container with id d721cc7cfd230de0adbba8bfea2c55a724815412002d2cafb83109af366873ee Oct 30 22:11:02.385464 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:02.385427 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:11:02.385575 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:02.385534 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:11:02.385640 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:02.385586 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:11:02.428799 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:02.428734 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" podStartSLOduration=2.428714408 podStartE2EDuration="2.428714408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:02.424277265 +0000 UTC m=+1264.976453565" watchObservedRunningTime="2026-10-30 22:11:02.428714408 +0000 UTC m=+1264.980890709" Oct 30 22:11:03.219275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:03.206179 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" event={"ID":"8033a54b-e69c-49d8-ad42-315c1486179a","Type":"ContainerStarted","Data":"8fe714b8954764a1a65c00861a029ef689ea1b8e7472eb57fa355401242fe705"} Oct 30 22:11:03.219275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:03.211820 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" event={"ID":"8033a54b-e69c-49d8-ad42-315c1486179a","Type":"ContainerStarted","Data":"d721cc7cfd230de0adbba8bfea2c55a724815412002d2cafb83109af366873ee"} Oct 30 22:11:03.277571 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:03.277531 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:11:03.277750 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:03.277616 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" podUID="bdbec1e1-b705-464b-88bd-6b6ddbfb3513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 22:11:03.320864 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:03.320721 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 22:11:03.320864 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:03.320801 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" podUID="d4a885f9-5fd5-4782-978b-ceb9fb740610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 22:11:03.367276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:03.366789 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" podStartSLOduration=2.366764549 podStartE2EDuration="2.366764549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:03.34367051 +0000 UTC m=+1265.895846811" watchObservedRunningTime="2026-10-30 22:11:03.366764549 +0000 UTC m=+1265.918940852" Oct 30 22:11:03.744274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:03.743469 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:11:03.977275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:03.976619 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:03.981273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:03.980797 6006 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:11:03.981273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:03.980865 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" podUID="8033a54b-e69c-49d8-ad42-315c1486179a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 22:11:04.286276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:04.278327 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:11:04.286276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:04.278447 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" podUID="bdbec1e1-b705-464b-88bd-6b6ddbfb3513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 22:11:04.327202 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:04.326381 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 22:11:04.327202 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:04.326469 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" podUID="d4a885f9-5fd5-4782-978b-ceb9fb740610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 22:11:04.979276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:04.978605 6006 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:11:04.979276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:04.978672 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" podUID="8033a54b-e69c-49d8-ad42-315c1486179a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 22:11:05.283337 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:05.277702 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:11:05.283337 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:05.277767 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" podUID="bdbec1e1-b705-464b-88bd-6b6ddbfb3513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 22:11:05.324283 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:05.323701 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 22:11:05.324283 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:05.323780 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" podUID="d4a885f9-5fd5-4782-978b-ceb9fb740610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 22:11:05.977337 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:05.976958 6006 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:11:05.977337 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:05.977028 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" podUID="8033a54b-e69c-49d8-ad42-315c1486179a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 22:11:06.278333 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:06.278133 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:11:06.278333 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:06.278198 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" podUID="bdbec1e1-b705-464b-88bd-6b6ddbfb3513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 22:11:06.326274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:06.325875 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 22:11:06.326274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:06.325935 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" podUID="d4a885f9-5fd5-4782-978b-ceb9fb740610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 22:11:06.993276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:06.992683 6006 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:11:06.993276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:06.992750 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" podUID="8033a54b-e69c-49d8-ad42-315c1486179a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 22:11:07.278915 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:07.278817 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:11:07.279177 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:07.279146 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" podUID="bdbec1e1-b705-464b-88bd-6b6ddbfb3513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 22:11:07.330277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:07.320903 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 22:11:07.330277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:07.320998 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" podUID="d4a885f9-5fd5-4782-978b-ceb9fb740610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 22:11:07.976759 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:07.976715 6006 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:11:07.978848 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:07.978809 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" podUID="8033a54b-e69c-49d8-ad42-315c1486179a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 22:11:08.301330 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:08.292692 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:11:08.301330 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:08.292770 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" podUID="bdbec1e1-b705-464b-88bd-6b6ddbfb3513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 22:11:08.324275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:08.318857 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 22:11:08.324275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:08.318918 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" podUID="d4a885f9-5fd5-4782-978b-ceb9fb740610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 22:11:08.977327 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:08.977284 6006 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:11:08.977756 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:08.977346 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" podUID="8033a54b-e69c-49d8-ad42-315c1486179a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 22:11:09.276928 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:09.276831 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:11:09.277076 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:09.276919 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" podUID="bdbec1e1-b705-464b-88bd-6b6ddbfb3513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 22:11:09.829579 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:09.829544 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:09.835014 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:09.834980 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:09.837056 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:09.837029 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:11:10.283755 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:10.283674 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:10.285758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:10.285728 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:10.286702 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:10.286669 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:11:10.424341 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:10.424309 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:10.424631 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:10.424612 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:10.425660 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:10.425609 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:11:27.912001 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:11:27.911960 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:11:43.068506 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:11:43.068468 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:11:57.915306 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:11:57.915260 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:12:17.821065 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:17.821026 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd"] Oct 30 22:12:17.825320 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:17.825299 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:17.925296 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:17.925215 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf"] Oct 30 22:12:17.930522 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:17.930484 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:17.992063 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:17.992013 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf"] Oct 30 22:12:17.992750 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:17.992723 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-workload-certs\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:17.992978 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:17.992761 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-data\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:17.992978 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:17.992793 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-token\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:17.992978 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:17.992823 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-847lh\" (UniqueName: \"kubernetes.io/projected/ff455cf0-f681-43de-b6b8-b33eb3c40562-kube-api-access-847lh\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:17.993089 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:17.992973 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff455cf0-f681-43de-b6b8-b33eb3c40562-istiod-ca-cert\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:17.993127 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:17.993109 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-workload-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:17.993196 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:17.993176 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-credential-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:17.993489 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:17.993467 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-podinfo\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:17.993668 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:17.993647 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-envoy\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:18.037016 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.036966 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd"] Oct 30 22:12:18.095235 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.095145 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-podinfo\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:18.095235 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.095188 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z7kh\" (UniqueName: \"kubernetes.io/projected/492acb10-fd86-4613-a1fc-68a0701b7846-kube-api-access-9z7kh\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.095235 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.095234 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-credential-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.095496 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.095286 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-envoy\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:18.095496 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.095447 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-workload-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.095584 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.095532 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-workload-certs\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:18.095584 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.095573 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-data\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:18.095673 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.095600 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-token\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:18.095673 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.095631 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-847lh\" (UniqueName: \"kubernetes.io/projected/ff455cf0-f681-43de-b6b8-b33eb3c40562-kube-api-access-847lh\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:18.096011 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.095984 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-workload-certs\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:18.096132 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.096063 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-istio-envoy\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.096132 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.096088 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-data\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:18.096385 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.096358 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff455cf0-f681-43de-b6b8-b33eb3c40562-istiod-ca-cert\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:18.096518 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.096397 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492acb10-fd86-4613-a1fc-68a0701b7846-istio-podinfo\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.096518 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.096439 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492acb10-fd86-4613-a1fc-68a0701b7846-istio-token\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.096518 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.096471 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492acb10-fd86-4613-a1fc-68a0701b7846-istiod-ca-cert\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.096681 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.096647 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-workload-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:18.096735 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.096688 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-workload-certs\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.096958 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.096941 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-workload-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:18.097064 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.097046 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-istio-data\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.097123 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.097094 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-credential-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:18.097413 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.097398 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-credential-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:18.097502 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.097451 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff455cf0-f681-43de-b6b8-b33eb3c40562-istiod-ca-cert\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:18.098201 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.098179 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-envoy\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:18.098201 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.098191 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-podinfo\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:18.154620 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.154592 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-token\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:18.154801 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.154777 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-847lh\" (UniqueName: \"kubernetes.io/projected/ff455cf0-f681-43de-b6b8-b33eb3c40562-kube-api-access-847lh\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:18.198535 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.198498 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-workload-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.198727 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.198545 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-istio-envoy\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.198727 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.198571 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492acb10-fd86-4613-a1fc-68a0701b7846-istio-podinfo\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.198727 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.198611 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492acb10-fd86-4613-a1fc-68a0701b7846-istio-token\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.198727 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.198643 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492acb10-fd86-4613-a1fc-68a0701b7846-istiod-ca-cert\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.198727 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.198718 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-workload-certs\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.198999 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.198755 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-istio-data\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.198999 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.198876 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z7kh\" (UniqueName: \"kubernetes.io/projected/492acb10-fd86-4613-a1fc-68a0701b7846-kube-api-access-9z7kh\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.198999 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.198940 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-credential-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.198999 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.198957 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-workload-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.199543 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.199028 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-workload-certs\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.199543 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.199307 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-istio-data\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.199725 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.199596 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492acb10-fd86-4613-a1fc-68a0701b7846-istiod-ca-cert\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.199725 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.199625 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-credential-socket\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.201603 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.201574 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-istio-envoy\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.201937 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.201916 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492acb10-fd86-4613-a1fc-68a0701b7846-istio-podinfo\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.257317 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.257281 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492acb10-fd86-4613-a1fc-68a0701b7846-istio-token\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.257471 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.257419 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z7kh\" (UniqueName: \"kubernetes.io/projected/492acb10-fd86-4613-a1fc-68a0701b7846-kube-api-access-9z7kh\") pod \"e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.439173 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.439080 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:18.550482 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.550444 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:18.866220 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.866188 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd"] Oct 30 22:12:18.866788 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:12:18.866762 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff455cf0_f681_43de_b6b8_b33eb3c40562.slice/crio-1ea7d68290b6b57c9b7a78ba9b80c0d1ec622c0273b4f4ffd640ff59c209b49d WatchSource:0}: Error finding container 1ea7d68290b6b57c9b7a78ba9b80c0d1ec622c0273b4f4ffd640ff59c209b49d: Status 404 returned error can't find the container with id 1ea7d68290b6b57c9b7a78ba9b80c0d1ec622c0273b4f4ffd640ff59c209b49d Oct 30 22:12:18.869442 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.869408 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:12:18.869531 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.869474 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:12:18.869531 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.869503 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:12:18.977324 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.977292 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf"] Oct 30 22:12:18.977731 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:12:18.977705 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492acb10_fd86_4613_a1fc_68a0701b7846.slice/crio-37b04c9bd43cc83beb59eccad432fdfdef7cb786ebd327b2a270733ee575423b WatchSource:0}: Error finding container 37b04c9bd43cc83beb59eccad432fdfdef7cb786ebd327b2a270733ee575423b: Status 404 returned error can't find the container with id 37b04c9bd43cc83beb59eccad432fdfdef7cb786ebd327b2a270733ee575423b Oct 30 22:12:18.980753 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.980709 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:12:18.980850 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.980791 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:12:18.980850 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:18.980832 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:12:19.878600 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:19.878559 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" event={"ID":"492acb10-fd86-4613-a1fc-68a0701b7846","Type":"ContainerStarted","Data":"50c3fe8a4021575af554f8504c4e041e1c32f8e77051a3555055129cf20d36e3"} Oct 30 22:12:19.879003 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:19.878624 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" event={"ID":"492acb10-fd86-4613-a1fc-68a0701b7846","Type":"ContainerStarted","Data":"37b04c9bd43cc83beb59eccad432fdfdef7cb786ebd327b2a270733ee575423b"} Oct 30 22:12:19.881157 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:19.881126 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" event={"ID":"ff455cf0-f681-43de-b6b8-b33eb3c40562","Type":"ContainerStarted","Data":"91e0f994561261eb2c2125848611ae1f727a40bc816f27d66a52dfec9f8fbe29"} Oct 30 22:12:19.881297 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:19.881183 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" event={"ID":"ff455cf0-f681-43de-b6b8-b33eb3c40562","Type":"ContainerStarted","Data":"1ea7d68290b6b57c9b7a78ba9b80c0d1ec622c0273b4f4ffd640ff59c209b49d"} Oct 30 22:12:20.045756 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:20.040719 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" podStartSLOduration=3.040695085 podStartE2EDuration="3.040695085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:19.998433794 +0000 UTC m=+1342.550610094" watchObservedRunningTime="2026-10-30 22:12:20.040695085 +0000 UTC m=+1342.592871388" Oct 30 22:12:20.154338 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:20.123030 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" podStartSLOduration=3.123010659 podStartE2EDuration="3.123010659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:20.121743614 +0000 UTC m=+1342.673919915" watchObservedRunningTime="2026-10-30 22:12:20.123010659 +0000 UTC m=+1342.675186959" Oct 30 22:12:20.443452 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:20.443326 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:20.444759 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:20.444727 6006 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 22:12:20.444998 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:20.444971 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" podUID="ff455cf0-f681-43de-b6b8-b33eb3c40562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 22:12:20.555840 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:20.555793 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:20.556537 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:20.556512 6006 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 22:12:20.556752 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:20.556728 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" podUID="492acb10-fd86-4613-a1fc-68a0701b7846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 22:12:21.447275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:21.446888 6006 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 22:12:21.447275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:21.446954 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" podUID="ff455cf0-f681-43de-b6b8-b33eb3c40562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 22:12:21.554323 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:21.553732 6006 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 22:12:21.554323 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:21.553799 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" podUID="492acb10-fd86-4613-a1fc-68a0701b7846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 22:12:22.442275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:22.441694 6006 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 22:12:22.442275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:22.441767 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" podUID="ff455cf0-f681-43de-b6b8-b33eb3c40562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 22:12:22.552086 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:22.552050 6006 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 22:12:22.552698 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:22.552665 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" podUID="492acb10-fd86-4613-a1fc-68a0701b7846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 22:12:23.450328 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:23.444123 6006 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 22:12:23.450328 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:23.444186 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" podUID="ff455cf0-f681-43de-b6b8-b33eb3c40562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 22:12:23.561139 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:23.560691 6006 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 22:12:23.561139 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:23.560751 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" podUID="492acb10-fd86-4613-a1fc-68a0701b7846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 22:12:24.454280 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:24.451034 6006 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 22:12:24.454280 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:24.451095 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" podUID="ff455cf0-f681-43de-b6b8-b33eb3c40562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 22:12:24.555274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:24.554670 6006 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 22:12:24.555274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:24.554735 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" podUID="492acb10-fd86-4613-a1fc-68a0701b7846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 22:12:25.440385 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:25.440340 6006 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 22:12:25.440867 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:25.440411 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" podUID="ff455cf0-f681-43de-b6b8-b33eb3c40562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 22:12:25.550891 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:25.550852 6006 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 22:12:25.551072 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:25.550914 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" podUID="492acb10-fd86-4613-a1fc-68a0701b7846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 22:12:26.703935 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:26.703886 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:26.705569 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:26.705542 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:26.706787 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:26.706760 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:12:26.809306 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:26.809271 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:26.809777 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:26.809754 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:26.811571 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:26.811504 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:12:27.912085 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:12:27.912049 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:12:29.811873 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:29.811830 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:12:42.482924 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:42.482839 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn"] Oct 30 22:12:42.488657 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:42.488628 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:12:42.577654 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:42.577616 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-home\") pod \"distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn\" (UID: \"08f28fb4-f013-4385-991d-742c9e38f4b2\") " pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:12:42.577836 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:42.577808 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-dshm\") pod \"distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn\" (UID: \"08f28fb4-f013-4385-991d-742c9e38f4b2\") " pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:12:42.577881 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:42.577853 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08f28fb4-f013-4385-991d-742c9e38f4b2-tls-certs\") pod \"distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn\" (UID: \"08f28fb4-f013-4385-991d-742c9e38f4b2\") " pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:12:42.577968 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:42.577945 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-tmp-dir\") pod \"distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn\" (UID: \"08f28fb4-f013-4385-991d-742c9e38f4b2\") " pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:12:42.578018 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:42.577990 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7m2r\" (UniqueName: \"kubernetes.io/projected/08f28fb4-f013-4385-991d-742c9e38f4b2-kube-api-access-h7m2r\") pod \"distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn\" (UID: \"08f28fb4-f013-4385-991d-742c9e38f4b2\") " pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:12:42.578116 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:42.578104 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-model-cache\") pod \"distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn\" (UID: \"08f28fb4-f013-4385-991d-742c9e38f4b2\") " pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:12:42.599188 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:42.599156 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-bc714d\"/\"distinct-2-w2-bc714d-kserve-self-signed-certs\"" Oct 30 22:12:42.602845 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:42.602813 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn"] Oct 30 22:12:42.681756 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:42.681718 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-dshm\") pod \"distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn\" (UID: \"08f28fb4-f013-4385-991d-742c9e38f4b2\") " pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:12:42.681756 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:42.681761 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08f28fb4-f013-4385-991d-742c9e38f4b2-tls-certs\") pod \"distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn\" (UID: \"08f28fb4-f013-4385-991d-742c9e38f4b2\") " pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:12:42.682027 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:42.681790 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-tmp-dir\") pod \"distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn\" (UID: \"08f28fb4-f013-4385-991d-742c9e38f4b2\") " pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:12:42.682027 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:42.681843 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7m2r\" (UniqueName: \"kubernetes.io/projected/08f28fb4-f013-4385-991d-742c9e38f4b2-kube-api-access-h7m2r\") pod \"distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn\" (UID: \"08f28fb4-f013-4385-991d-742c9e38f4b2\") " pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:12:42.682027 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:42.681901 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-model-cache\") pod \"distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn\" (UID: \"08f28fb4-f013-4385-991d-742c9e38f4b2\") " pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:12:42.682027 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:42.682007 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-home\") pod \"distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn\" (UID: \"08f28fb4-f013-4385-991d-742c9e38f4b2\") " pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:12:42.683600 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:42.683264 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-tmp-dir\") pod \"distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn\" (UID: \"08f28fb4-f013-4385-991d-742c9e38f4b2\") " pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:12:42.683600 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:42.683322 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-home\") pod \"distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn\" (UID: \"08f28fb4-f013-4385-991d-742c9e38f4b2\") " pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:12:42.683600 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:42.683541 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-model-cache\") pod \"distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn\" (UID: \"08f28fb4-f013-4385-991d-742c9e38f4b2\") " pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:12:42.695860 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:42.692052 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-dshm\") pod \"distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn\" (UID: \"08f28fb4-f013-4385-991d-742c9e38f4b2\") " pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:12:42.696541 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:42.696510 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08f28fb4-f013-4385-991d-742c9e38f4b2-tls-certs\") pod \"distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn\" (UID: \"08f28fb4-f013-4385-991d-742c9e38f4b2\") " pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:12:42.743342 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:42.742486 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7m2r\" (UniqueName: \"kubernetes.io/projected/08f28fb4-f013-4385-991d-742c9e38f4b2-kube-api-access-h7m2r\") pod \"distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn\" (UID: \"08f28fb4-f013-4385-991d-742c9e38f4b2\") " pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:12:42.811418 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:42.811374 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:12:43.305229 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:12:43.305189 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08f28fb4_f013_4385_991d_742c9e38f4b2.slice/crio-194db9587d5f1243135e553a2caae2b1b1e19ae7783a0678a2481ffd70294109 WatchSource:0}: Error finding container 194db9587d5f1243135e553a2caae2b1b1e19ae7783a0678a2481ffd70294109: Status 404 returned error can't find the container with id 194db9587d5f1243135e553a2caae2b1b1e19ae7783a0678a2481ffd70294109 Oct 30 22:12:43.305536 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:43.305508 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn"] Oct 30 22:12:44.156323 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:44.156245 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" event={"ID":"08f28fb4-f013-4385-991d-742c9e38f4b2","Type":"ContainerStarted","Data":"947682e4a57dd01bffdc20aab0a087343fca49ca45f8637b1064679504cc061e"} Oct 30 22:12:44.156323 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:44.156313 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" event={"ID":"08f28fb4-f013-4385-991d-742c9e38f4b2","Type":"ContainerStarted","Data":"194db9587d5f1243135e553a2caae2b1b1e19ae7783a0678a2481ffd70294109"} Oct 30 22:12:44.272335 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:44.272274 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" podStartSLOduration=2.272229209 podStartE2EDuration="2.272229209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:44.271367929 +0000 UTC m=+1366.823544228" watchObservedRunningTime="2026-10-30 22:12:44.272229209 +0000 UTC m=+1366.824405509" Oct 30 22:12:44.812380 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:44.812342 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:12:52.812463 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:52.812412 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:12:52.825977 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:52.825939 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:12:53.245162 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:53.245080 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:12:56.074666 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:56.074624 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2c023d8-mg7lg"] Oct 30 22:12:56.122570 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:56.122523 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2c023d8-mg7lg"] Oct 30 22:12:57.606410 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:12:57.606371 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d09cbbc8-a913-47a1-aee5-c9ee6771abe8" path="/var/lib/kubelet/pods/d09cbbc8-a913-47a1-aee5-c9ee6771abe8/volumes" Oct 30 22:12:57.928778 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:12:57.928663 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:13:00.261413 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:00.261373 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt"] Oct 30 22:13:00.296298 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:00.291464 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:13:00.493344 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:00.493306 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt"] Oct 30 22:13:00.495293 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:00.494110 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c9dbe6\"/\"distinct-2-w5-c9dbe6-kserve-self-signed-certs\"" Oct 30 22:13:00.521372 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:00.496142 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-tmp-dir\") pod \"distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt\" (UID: \"4e5d742b-c020-45ff-bd02-76663b0bb04a\") " pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:13:00.521372 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:00.496307 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-model-cache\") pod \"distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt\" (UID: \"4e5d742b-c020-45ff-bd02-76663b0bb04a\") " pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:13:00.521372 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:00.496354 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-dshm\") pod \"distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt\" (UID: \"4e5d742b-c020-45ff-bd02-76663b0bb04a\") " pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:13:00.521372 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:00.496409 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79m9b\" (UniqueName: \"kubernetes.io/projected/4e5d742b-c020-45ff-bd02-76663b0bb04a-kube-api-access-79m9b\") pod \"distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt\" (UID: \"4e5d742b-c020-45ff-bd02-76663b0bb04a\") " pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:13:00.521372 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:00.496523 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-home\") pod \"distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt\" (UID: \"4e5d742b-c020-45ff-bd02-76663b0bb04a\") " pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:13:00.521372 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:00.496605 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e5d742b-c020-45ff-bd02-76663b0bb04a-tls-certs\") pod \"distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt\" (UID: \"4e5d742b-c020-45ff-bd02-76663b0bb04a\") " pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:13:00.611915 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:00.611779 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-dshm\") pod \"distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt\" (UID: \"4e5d742b-c020-45ff-bd02-76663b0bb04a\") " pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:13:00.612264 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:00.612222 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79m9b\" (UniqueName: \"kubernetes.io/projected/4e5d742b-c020-45ff-bd02-76663b0bb04a-kube-api-access-79m9b\") pod \"distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt\" (UID: \"4e5d742b-c020-45ff-bd02-76663b0bb04a\") " pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:13:00.612530 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:00.612509 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-home\") pod \"distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt\" (UID: \"4e5d742b-c020-45ff-bd02-76663b0bb04a\") " pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:13:00.612746 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:00.612724 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e5d742b-c020-45ff-bd02-76663b0bb04a-tls-certs\") pod \"distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt\" (UID: \"4e5d742b-c020-45ff-bd02-76663b0bb04a\") " pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:13:00.612890 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:00.612872 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-tmp-dir\") pod \"distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt\" (UID: \"4e5d742b-c020-45ff-bd02-76663b0bb04a\") " pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:13:00.613153 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:00.613131 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-model-cache\") pod \"distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt\" (UID: \"4e5d742b-c020-45ff-bd02-76663b0bb04a\") " pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:13:00.613803 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:00.613778 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-model-cache\") pod \"distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt\" (UID: \"4e5d742b-c020-45ff-bd02-76663b0bb04a\") " pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:13:00.618571 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:00.617192 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-tmp-dir\") pod \"distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt\" (UID: \"4e5d742b-c020-45ff-bd02-76663b0bb04a\") " pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:13:00.618571 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:00.617831 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-home\") pod \"distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt\" (UID: \"4e5d742b-c020-45ff-bd02-76663b0bb04a\") " pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:13:00.618571 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:00.618425 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e5d742b-c020-45ff-bd02-76663b0bb04a-tls-certs\") pod \"distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt\" (UID: \"4e5d742b-c020-45ff-bd02-76663b0bb04a\") " pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:13:00.670795 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:00.670596 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-dshm\") pod \"distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt\" (UID: \"4e5d742b-c020-45ff-bd02-76663b0bb04a\") " pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:13:00.711942 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:00.711905 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79m9b\" (UniqueName: \"kubernetes.io/projected/4e5d742b-c020-45ff-bd02-76663b0bb04a-kube-api-access-79m9b\") pod \"distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt\" (UID: \"4e5d742b-c020-45ff-bd02-76663b0bb04a\") " pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:13:00.924865 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:00.924833 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:13:01.123228 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:01.122410 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:01.256357 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:01.250355 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:01.300324 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:01.299374 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:01.953403 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:01.945660 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:02.056919 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:02.039446 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:02.138334 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:02.132545 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 22:13:02.160577 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:02.160529 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 22:13:02.319121 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:02.319057 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:02.515210 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:02.441642 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:02.515210 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:02.500587 6006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Oct 30 22:13:02.554614 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:02.554576 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:13:02.580964 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:02.580925 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:13:02.894391 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:02.864436 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt"] Oct 30 22:13:03.091345 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:02.998378 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:03.091345 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:02.999629 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:03.091345 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:03.000850 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:13:03.091345 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:03.040342 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:03.091345 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:03.041901 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:13:03.091345 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:03.058386 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:03.185275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:03.146794 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:03.185639 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:03.169643 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:03.197496 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:03.197443 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:13:03.307276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:03.286954 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 22:13:03.331916 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:03.331854 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 22:13:03.340290 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:03.336336 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:13:03.355337 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:03.345809 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" event={"ID":"4e5d742b-c020-45ff-bd02-76663b0bb04a","Type":"ContainerStarted","Data":"119ed64f7de1e69ffc1364843fc873429a4b803a04eca78e18e7ad22a55afbe4"} Oct 30 22:13:03.436837 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:03.428531 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:13:03.628145 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:03.628048 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:13:03.653367 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:03.651946 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:13:03.896348 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:03.890085 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:13:03.942275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:03.925339 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:13:04.379270 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:04.378968 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" event={"ID":"4e5d742b-c020-45ff-bd02-76663b0bb04a","Type":"ContainerStarted","Data":"4b183b869e530fb82a87326affa9db7c50a8780269d475c2c6f6375d9d43ff5d"} Oct 30 22:13:04.928209 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:04.925903 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:13:10.925853 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:10.925812 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:13:10.950567 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:10.950534 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:13:11.067833 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:11.067770 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" podStartSLOduration=11.06775419 podStartE2EDuration="11.06775419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:04.664703845 +0000 UTC m=+1387.216880150" watchObservedRunningTime="2026-10-30 22:13:11.06775419 +0000 UTC m=+1393.619930520" Oct 30 22:13:11.453438 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:11.453392 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:13:12.057199 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:12.057163 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b293ccec-62fvd"] Oct 30 22:13:12.065511 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:12.065476 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b293ccec-62fvd"] Oct 30 22:13:13.607094 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:13.607053 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86404af4-977e-4617-ab0d-18a350387d01" path="/var/lib/kubelet/pods/86404af4-977e-4617-ab0d-18a350387d01/volumes" Oct 30 22:13:17.820944 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:17.820907 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9"] Oct 30 22:13:17.826433 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:17.826410 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:17.923610 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:17.923563 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istiod-ca-cert\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:17.923610 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:17.923612 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-podinfo\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:17.923934 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:17.923759 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-envoy\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:17.923934 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:17.923821 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-token\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:17.924022 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:17.924006 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-workload-certs\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:17.924062 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:17.924036 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-data\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:17.924187 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:17.924172 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-credential-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:17.924274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:17.924207 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml8lm\" (UniqueName: \"kubernetes.io/projected/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-kube-api-access-ml8lm\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:17.924274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:17.924240 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-workload-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:18.025794 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.025751 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-workload-certs\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:18.026160 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.026127 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-data\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:18.026422 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.026264 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-credential-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:18.026422 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.026297 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml8lm\" (UniqueName: \"kubernetes.io/projected/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-kube-api-access-ml8lm\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:18.026422 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.026329 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-workload-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:18.026422 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.026415 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istiod-ca-cert\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:18.026750 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.026447 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-podinfo\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:18.026750 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.026491 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-envoy\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:18.026750 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.026542 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-token\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:18.026750 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.026773 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-workload-certs\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:18.033350 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.027305 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-credential-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:18.033548 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.027853 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-workload-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:18.033670 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.028403 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istiod-ca-cert\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:18.033775 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.030146 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-data\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:18.033885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.031244 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-podinfo\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:18.034017 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.032494 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-envoy\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:18.662294 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.662238 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn"] Oct 30 22:13:18.667734 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.667704 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2"] Oct 30 22:13:18.667926 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.667843 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.672212 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.672187 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9"] Oct 30 22:13:18.672367 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.672290 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:18.713909 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.713874 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-token\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:18.714093 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.713928 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml8lm\" (UniqueName: \"kubernetes.io/projected/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-kube-api-access-ml8lm\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:18.714524 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.714507 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn"] Oct 30 22:13:18.717741 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.717719 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2"] Oct 30 22:13:18.720438 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.720421 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5274f32b-2ph2b"] Oct 30 22:13:18.722936 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.722913 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5274f32b-2ph2b"] Oct 30 22:13:18.735771 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.735737 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-podinfo\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.735925 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.735905 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-data\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.735995 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.735960 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-credential-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.736105 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.736080 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-workload-certs\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.736233 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.736213 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-token\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.736381 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.736356 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-envoy\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.736453 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.736400 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4h8l\" (UniqueName: \"kubernetes.io/projected/67ba26ec-c30a-46f7-8b3d-d6ef88849246-kube-api-access-z4h8l\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.736511 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.736494 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istiod-ca-cert\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.736645 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.736623 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-workload-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.742433 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.742392 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:18.837813 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.837754 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-token\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:18.837813 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.837815 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-envoy\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.839151 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.837846 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4h8l\" (UniqueName: \"kubernetes.io/projected/67ba26ec-c30a-46f7-8b3d-d6ef88849246-kube-api-access-z4h8l\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.839151 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.837894 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istiod-ca-cert\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.839151 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.837940 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-data\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:18.839151 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.837998 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-workload-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.839151 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.838072 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-podinfo\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.839151 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.838099 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-envoy\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:18.839151 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.838190 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-workload-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:18.839151 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.838244 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbtlw\" (UniqueName: \"kubernetes.io/projected/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-kube-api-access-zbtlw\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:18.839151 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.838326 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-podinfo\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:18.839151 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.838352 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-data\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.839151 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.838404 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-credential-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.839151 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.838470 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-workload-certs\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.839151 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.838513 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-credential-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:18.839151 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.838554 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-workload-certs\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:18.839151 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.838609 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-token\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.839151 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.838697 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istiod-ca-cert\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:18.840324 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.839738 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istiod-ca-cert\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.840324 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.839779 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-data\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.840324 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.839814 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-workload-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.841491 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.841343 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-credential-socket\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.841584 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.841517 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-workload-certs\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.843904 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.843861 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-envoy\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.847329 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.847217 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-podinfo\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.904659 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.904625 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-token\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.939607 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.939517 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-credential-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:18.939607 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.939568 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-workload-certs\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:18.939822 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.939632 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istiod-ca-cert\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:18.939822 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.939656 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-token\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:18.939822 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.939698 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-data\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:18.939822 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.939768 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-envoy\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:18.940034 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.939852 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-workload-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:18.940034 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.939904 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbtlw\" (UniqueName: \"kubernetes.io/projected/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-kube-api-access-zbtlw\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:18.940034 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.940012 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-podinfo\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:18.940203 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.940065 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-workload-certs\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:18.940453 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.940427 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istiod-ca-cert\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:18.940603 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.940584 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-workload-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:18.940603 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.940596 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-data\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:18.940698 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.940673 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-credential-socket\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:18.942401 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.942372 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-envoy\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:18.942870 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.942850 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-podinfo\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:18.957209 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.957161 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4h8l\" (UniqueName: \"kubernetes.io/projected/67ba26ec-c30a-46f7-8b3d-d6ef88849246-kube-api-access-z4h8l\") pod \"e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:18.984508 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:18.984464 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:19.014726 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:19.014682 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbtlw\" (UniqueName: \"kubernetes.io/projected/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-kube-api-access-zbtlw\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:19.017032 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:19.017009 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-token\") pod \"e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:19.175215 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:19.175184 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9"] Oct 30 22:13:19.176527 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:13:19.176494 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab56f9ed_7e2f_4a74_8dda_53df36c32f23.slice/crio-2c6454804872d766de22f5a2c3c59cc538051be1a76df198942c134701152f8e WatchSource:0}: Error finding container 2c6454804872d766de22f5a2c3c59cc538051be1a76df198942c134701152f8e: Status 404 returned error can't find the container with id 2c6454804872d766de22f5a2c3c59cc538051be1a76df198942c134701152f8e Oct 30 22:13:19.179367 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:19.179329 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:13:19.179447 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:19.179401 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:13:19.179447 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:19.179433 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:13:19.291113 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:19.291074 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:19.394592 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:13:19.386600 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ba26ec_c30a_46f7_8b3d_d6ef88849246.slice/crio-b9956bc7e2ebc99a0d5965a6faa759071d1c9ef2b6cf8bbc8ed41a61cb1eda83 WatchSource:0}: Error finding container b9956bc7e2ebc99a0d5965a6faa759071d1c9ef2b6cf8bbc8ed41a61cb1eda83: Status 404 returned error can't find the container with id b9956bc7e2ebc99a0d5965a6faa759071d1c9ef2b6cf8bbc8ed41a61cb1eda83 Oct 30 22:13:19.394592 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:19.392165 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn"] Oct 30 22:13:19.412561 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:19.411039 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:13:19.412561 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:19.411142 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:13:19.412561 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:19.411191 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:13:19.509526 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:19.509487 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" event={"ID":"ab56f9ed-7e2f-4a74-8dda-53df36c32f23","Type":"ContainerStarted","Data":"a4885693acc976444deef6cd022219f61bd92b56374102a38907c95f2a0502a1"} Oct 30 22:13:19.509781 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:19.509555 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" event={"ID":"ab56f9ed-7e2f-4a74-8dda-53df36c32f23","Type":"ContainerStarted","Data":"2c6454804872d766de22f5a2c3c59cc538051be1a76df198942c134701152f8e"} Oct 30 22:13:19.512416 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:19.512345 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" event={"ID":"67ba26ec-c30a-46f7-8b3d-d6ef88849246","Type":"ContainerStarted","Data":"b9956bc7e2ebc99a0d5965a6faa759071d1c9ef2b6cf8bbc8ed41a61cb1eda83"} Oct 30 22:13:19.607090 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:19.607054 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a8cc94b-62fb-4406-b0b4-bc383a2c892f" path="/var/lib/kubelet/pods/6a8cc94b-62fb-4406-b0b4-bc383a2c892f/volumes" Oct 30 22:13:19.645897 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:19.645830 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" podStartSLOduration=2.645809056 podStartE2EDuration="2.645809056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:19.64098802 +0000 UTC m=+1402.193164340" watchObservedRunningTime="2026-10-30 22:13:19.645809056 +0000 UTC m=+1402.197985357" Oct 30 22:13:19.684700 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:19.684669 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2"] Oct 30 22:13:19.685762 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:13:19.685733 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda852f49d_5ed1_472d_8fb0_f811bbff7e4f.slice/crio-dab45a25adab0b275b28b4948b6f74e40f5ad6bfe7a134e1a7c89eac48580669 WatchSource:0}: Error finding container dab45a25adab0b275b28b4948b6f74e40f5ad6bfe7a134e1a7c89eac48580669: Status 404 returned error can't find the container with id dab45a25adab0b275b28b4948b6f74e40f5ad6bfe7a134e1a7c89eac48580669 Oct 30 22:13:19.689457 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:19.689413 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:13:19.689604 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:19.689524 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:13:19.689604 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:19.689558 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:13:19.743461 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:19.743431 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:20.541639 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:20.541586 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" event={"ID":"a852f49d-5ed1-472d-8fb0-f811bbff7e4f","Type":"ContainerStarted","Data":"1070d264a2421a18dfb291e0a9ba378427eef18cc6a07b8df53c4badd4b28a2e"} Oct 30 22:13:20.542158 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:20.541664 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" event={"ID":"a852f49d-5ed1-472d-8fb0-f811bbff7e4f","Type":"ContainerStarted","Data":"dab45a25adab0b275b28b4948b6f74e40f5ad6bfe7a134e1a7c89eac48580669"} Oct 30 22:13:20.544778 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:20.544739 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" event={"ID":"67ba26ec-c30a-46f7-8b3d-d6ef88849246","Type":"ContainerStarted","Data":"28f04748c2990fc6b8c23ed57f6df2026b60a46591783d8fe22d8c2e156df2f0"} Oct 30 22:13:20.692032 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:20.691955 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" podStartSLOduration=3.691936418 podStartE2EDuration="3.691936418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:20.682762185 +0000 UTC m=+1403.234938485" watchObservedRunningTime="2026-10-30 22:13:20.691936418 +0000 UTC m=+1403.244112784" Oct 30 22:13:20.754430 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:20.754389 6006 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 22:13:20.754707 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:20.754678 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" podUID="ab56f9ed-7e2f-4a74-8dda-53df36c32f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 22:13:20.806997 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:20.806855 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" podStartSLOduration=3.806831892 podStartE2EDuration="3.806831892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:20.803892117 +0000 UTC m=+1403.356068420" watchObservedRunningTime="2026-10-30 22:13:20.806831892 +0000 UTC m=+1403.359008199" Oct 30 22:13:20.992274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:20.991418 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:21.030290 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:21.018373 6006 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:13:21.030290 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:21.018464 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" podUID="67ba26ec-c30a-46f7-8b3d-d6ef88849246" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 22:13:21.299746 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:21.297686 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:21.317176 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:21.317138 6006 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 22:13:21.317525 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:21.317491 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" podUID="a852f49d-5ed1-472d-8fb0-f811bbff7e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 22:13:21.761347 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:21.755943 6006 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 22:13:21.761347 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:21.756009 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" podUID="ab56f9ed-7e2f-4a74-8dda-53df36c32f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 22:13:21.994744 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:21.994709 6006 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:13:21.995059 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:21.995010 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" podUID="67ba26ec-c30a-46f7-8b3d-d6ef88849246" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 22:13:22.291551 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:22.291506 6006 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 22:13:22.291724 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:22.291571 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" podUID="a852f49d-5ed1-472d-8fb0-f811bbff7e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 22:13:22.631334 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:22.630806 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f16663a3-htn9q"] Oct 30 22:13:22.662288 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:22.662239 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16663a3-htn9q" Oct 30 22:13:22.738276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:22.713627 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f16663a3-htn9q"] Oct 30 22:13:22.738606 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:22.726662 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:13:22.765402 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:22.726827 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:13:22.765945 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:22.745089 6006 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 22:13:22.778382 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:22.778335 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" podUID="ab56f9ed-7e2f-4a74-8dda-53df36c32f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 22:13:22.805492 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:22.746524 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68086ae0-79c5-403b-bebb-03ae08256b41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f16663a3-htn9q\" (UID: \"68086ae0-79c5-403b-bebb-03ae08256b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16663a3-htn9q" Oct 30 22:13:22.805949 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:22.805922 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gzqp\" (UniqueName: \"kubernetes.io/projected/68086ae0-79c5-403b-bebb-03ae08256b41-kube-api-access-5gzqp\") pod \"maas-api-revoke-sub-f16663a3-htn9q\" (UID: \"68086ae0-79c5-403b-bebb-03ae08256b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16663a3-htn9q" Oct 30 22:13:22.830274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:22.806427 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68086ae0-79c5-403b-bebb-03ae08256b41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f16663a3-htn9q\" (UID: \"68086ae0-79c5-403b-bebb-03ae08256b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16663a3-htn9q" Oct 30 22:13:22.991852 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:22.991747 6006 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:13:22.992163 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:22.992131 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" podUID="67ba26ec-c30a-46f7-8b3d-d6ef88849246" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 22:13:23.024276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:23.017358 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68086ae0-79c5-403b-bebb-03ae08256b41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f16663a3-htn9q\" (UID: \"68086ae0-79c5-403b-bebb-03ae08256b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16663a3-htn9q" Oct 30 22:13:23.032361 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:23.028500 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68086ae0-79c5-403b-bebb-03ae08256b41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f16663a3-htn9q\" (UID: \"68086ae0-79c5-403b-bebb-03ae08256b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16663a3-htn9q" Oct 30 22:13:23.298511 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:23.077073 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gzqp\" (UniqueName: \"kubernetes.io/projected/68086ae0-79c5-403b-bebb-03ae08256b41-kube-api-access-5gzqp\") pod \"maas-api-revoke-sub-f16663a3-htn9q\" (UID: \"68086ae0-79c5-403b-bebb-03ae08256b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16663a3-htn9q" Oct 30 22:13:23.360360 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:23.331600 6006 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 22:13:23.360360 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:23.331666 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" podUID="a852f49d-5ed1-472d-8fb0-f811bbff7e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 22:13:23.439277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:23.414113 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68086ae0-79c5-403b-bebb-03ae08256b41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f16663a3-htn9q\" (UID: \"68086ae0-79c5-403b-bebb-03ae08256b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16663a3-htn9q" Oct 30 22:13:23.670014 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:23.669884 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:23.763777 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:23.759834 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68086ae0-79c5-403b-bebb-03ae08256b41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f16663a3-htn9q\" (UID: \"68086ae0-79c5-403b-bebb-03ae08256b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16663a3-htn9q" Oct 30 22:13:23.886282 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:23.854116 6006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": context deadline exceeded" Oct 30 22:13:23.886282 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:23.854816 6006 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 22:13:23.886282 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:23.854895 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" podUID="ab56f9ed-7e2f-4a74-8dda-53df36c32f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 22:13:23.940198 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:23.940085 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:23.960871 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:23.944757 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gzqp\" (UniqueName: \"kubernetes.io/projected/68086ae0-79c5-403b-bebb-03ae08256b41-kube-api-access-5gzqp\") pod \"maas-api-revoke-sub-f16663a3-htn9q\" (UID: \"68086ae0-79c5-403b-bebb-03ae08256b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16663a3-htn9q" Oct 30 22:13:23.987275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:23.987026 6006 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:13:23.987275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:23.987089 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" podUID="67ba26ec-c30a-46f7-8b3d-d6ef88849246" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 22:13:24.241658 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:24.241554 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16663a3-htn9q" Oct 30 22:13:24.243634 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:24.243595 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:24.255681 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:24.244600 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:24.298890 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:24.298421 6006 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 22:13:24.298890 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:24.298485 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" podUID="a852f49d-5ed1-472d-8fb0-f811bbff7e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 22:13:24.530342 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:24.499147 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:24.705432 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:24.704029 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 22:13:24.705432 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:24.704441 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:13:24.721400 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:24.721337 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:24.787509 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:24.787403 6006 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 22:13:24.787794 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:24.787763 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" podUID="ab56f9ed-7e2f-4a74-8dda-53df36c32f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 22:13:25.003346 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:24.999606 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 22:13:25.218451 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:25.184718 6006 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:13:25.218451 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:25.184773 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" podUID="67ba26ec-c30a-46f7-8b3d-d6ef88849246" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 22:13:25.218451 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:25.061025 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:25.218451 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:25.049856 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 22:13:25.218451 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:25.185789 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:13:25.303192 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:25.303144 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:25.482596 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:25.482485 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:25.520796 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:25.520759 6006 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 22:13:25.521056 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:25.521030 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" podUID="a852f49d-5ed1-472d-8fb0-f811bbff7e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 22:13:25.567283 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:25.560657 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 22:13:25.576769 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:25.576717 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:25.814356 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:25.804760 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:13:25.931985 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:25.322204 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:25.955057 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:25.955019 6006 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 22:13:26.009049 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.009004 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" podUID="ab56f9ed-7e2f-4a74-8dda-53df36c32f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 22:13:26.170635 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.140755 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:13:26.170635 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.160154 6006 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:13:26.170635 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.160214 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" podUID="67ba26ec-c30a-46f7-8b3d-d6ef88849246" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 22:13:26.216336 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.178355 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 22:13:26.216336 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.179701 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:26.216336 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.187144 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:13:26.216336 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.188470 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:13:26.216695 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.216587 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:26.216988 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.216953 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 22:13:26.217279 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.217239 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 22:13:26.292585 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.292185 6006 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 22:13:26.292585 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.292264 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" podUID="a852f49d-5ed1-472d-8fb0-f811bbff7e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 22:13:26.492353 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.492322 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f16663a3-htn9q"] Oct 30 22:13:26.492640 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.492620 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:13:26.492808 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.492794 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:13:26.492921 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.492912 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:13:26.493024 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.493016 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:13:26.493128 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.493118 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:13:26.493238 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.493228 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:13:26.493377 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.493368 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:13:26.493474 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.493466 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:13:26.493903 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.493557 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:13:26.494814 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.494657 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:26.495323 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.495300 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:13:26.495426 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.495409 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:13:26.653951 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.653915 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:13:26.693330 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.693301 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:13:26.767019 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.766953 6006 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 22:13:26.767208 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.767014 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" podUID="ab56f9ed-7e2f-4a74-8dda-53df36c32f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 22:13:26.986500 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.986455 6006 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:13:26.986854 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:26.986525 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" podUID="67ba26ec-c30a-46f7-8b3d-d6ef88849246" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 22:13:27.292816 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:27.292699 6006 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 22:13:27.292816 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:27.292768 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" podUID="a852f49d-5ed1-472d-8fb0-f811bbff7e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 22:13:27.373220 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:27.373177 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16663a3-htn9q" event={"ID":"68086ae0-79c5-403b-bebb-03ae08256b41","Type":"ContainerStarted","Data":"13e37090876081c28c8b32773cfc61058e3608618f856670b3189a0d24ab6798"} Oct 30 22:13:27.373544 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:27.373516 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16663a3-htn9q" event={"ID":"68086ae0-79c5-403b-bebb-03ae08256b41","Type":"ContainerStarted","Data":"fdbc9e4b65e55366e8ad18fbd8ffe2e1c8c67b32710d5e0b263e97d4d3a4f5d3"} Oct 30 22:13:28.092353 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:13:28.091435 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:13:28.203330 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:28.196165 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16663a3-htn9q" podStartSLOduration=6.196143041 podStartE2EDuration="6.196143041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:27.964985099 +0000 UTC m=+1410.517161397" watchObservedRunningTime="2026-10-30 22:13:28.196143041 +0000 UTC m=+1410.748319342" Oct 30 22:13:28.229301 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:28.227925 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:28.238372 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:28.233751 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:28.263274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:28.252682 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:13:28.483159 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:28.482650 6006 generic.go:356] "Generic (PLEG): container finished" podID="68086ae0-79c5-403b-bebb-03ae08256b41" containerID="13e37090876081c28c8b32773cfc61058e3608618f856670b3189a0d24ab6798" exitCode=0 Oct 30 22:13:28.484526 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:28.484192 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16663a3-htn9q" event={"ID":"68086ae0-79c5-403b-bebb-03ae08256b41","Type":"ContainerDied","Data":"13e37090876081c28c8b32773cfc61058e3608618f856670b3189a0d24ab6798"} Oct 30 22:13:28.678628 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:28.678590 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:28.681082 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:28.680169 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:28.684329 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:28.683815 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:13:28.756931 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:28.756843 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:28.757506 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:28.757486 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:28.759140 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:28.759116 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:13:29.052101 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:29.052057 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcca0367-cdxzf"] Oct 30 22:13:29.060387 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:29.060345 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcca0367-cdxzf"] Oct 30 22:13:29.608781 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:29.608745 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bc00cea-a69e-4e48-90ef-c7a147696db8" path="/var/lib/kubelet/pods/1bc00cea-a69e-4e48-90ef-c7a147696db8/volumes" Oct 30 22:13:29.681830 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:29.681802 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16663a3-htn9q" Oct 30 22:13:29.729183 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:29.729146 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68086ae0-79c5-403b-bebb-03ae08256b41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68086ae0-79c5-403b-bebb-03ae08256b41-maas-api-openshift-ca\") pod \"68086ae0-79c5-403b-bebb-03ae08256b41\" (UID: \"68086ae0-79c5-403b-bebb-03ae08256b41\") " Oct 30 22:13:29.729405 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:29.729214 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68086ae0-79c5-403b-bebb-03ae08256b41-kube-api-access-5gzqp\" (UniqueName: \"kubernetes.io/projected/68086ae0-79c5-403b-bebb-03ae08256b41-kube-api-access-5gzqp\") pod \"68086ae0-79c5-403b-bebb-03ae08256b41\" (UID: \"68086ae0-79c5-403b-bebb-03ae08256b41\") " Oct 30 22:13:29.729473 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:29.729445 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68086ae0-79c5-403b-bebb-03ae08256b41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68086ae0-79c5-403b-bebb-03ae08256b41-maas-api-xks-ca\") pod \"68086ae0-79c5-403b-bebb-03ae08256b41\" (UID: \"68086ae0-79c5-403b-bebb-03ae08256b41\") " Oct 30 22:13:29.729702 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:29.729673 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68086ae0-79c5-403b-bebb-03ae08256b41-maas-api-openshift-ca" pod "68086ae0-79c5-403b-bebb-03ae08256b41" (UID: "68086ae0-79c5-403b-bebb-03ae08256b41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:29.730087 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:29.730070 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68086ae0-79c5-403b-bebb-03ae08256b41-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:29.732420 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:29.732387 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68086ae0-79c5-403b-bebb-03ae08256b41-kube-api-access-5gzqp" pod "68086ae0-79c5-403b-bebb-03ae08256b41" (UID: "68086ae0-79c5-403b-bebb-03ae08256b41"). InnerVolumeSpecName "kube-api-access-5gzqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:29.732744 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:29.732721 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68086ae0-79c5-403b-bebb-03ae08256b41-maas-api-xks-ca" pod "68086ae0-79c5-403b-bebb-03ae08256b41" (UID: "68086ae0-79c5-403b-bebb-03ae08256b41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:13:29.831098 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:29.831062 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gzqp\" (UniqueName: \"kubernetes.io/projected/68086ae0-79c5-403b-bebb-03ae08256b41-kube-api-access-5gzqp\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:29.831098 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:29.831091 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68086ae0-79c5-403b-bebb-03ae08256b41-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:30.498769 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:30.498733 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16663a3-htn9q" Oct 30 22:13:30.498769 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:30.498752 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16663a3-htn9q" event={"ID":"68086ae0-79c5-403b-bebb-03ae08256b41","Type":"ContainerDied","Data":"fdbc9e4b65e55366e8ad18fbd8ffe2e1c8c67b32710d5e0b263e97d4d3a4f5d3"} Oct 30 22:13:30.498998 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:30.498808 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdbc9e4b65e55366e8ad18fbd8ffe2e1c8c67b32710d5e0b263e97d4d3a4f5d3" Oct 30 22:13:30.663343 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:30.663296 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88af5db5-v5rds"] Oct 30 22:13:30.664289 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:30.664261 6006 cpu_manager.go:414] "containerMap: removing container" podUID="68086ae0-79c5-403b-bebb-03ae08256b41" containerName="revoke-keys" Oct 30 22:13:30.664289 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:30.664279 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68086ae0-79c5-403b-bebb-03ae08256b41" containerName="revoke-keys" Oct 30 22:13:30.664489 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:30.664476 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="68086ae0-79c5-403b-bebb-03ae08256b41" containerName="revoke-keys" Oct 30 22:13:30.668853 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:30.668818 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af5db5-v5rds" Oct 30 22:13:30.719964 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:30.719920 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88af5db5-v5rds"] Oct 30 22:13:30.728862 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:30.728544 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:13:30.731315 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:30.729189 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:13:30.748363 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:30.748318 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dw5l\" (UniqueName: \"kubernetes.io/projected/b03b6525-1963-4875-91d9-025d1e884994-kube-api-access-8dw5l\") pod \"maas-api-revoke-sub-88af5db5-v5rds\" (UID: \"b03b6525-1963-4875-91d9-025d1e884994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af5db5-v5rds" Oct 30 22:13:30.748599 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:30.748571 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b03b6525-1963-4875-91d9-025d1e884994-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88af5db5-v5rds\" (UID: \"b03b6525-1963-4875-91d9-025d1e884994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af5db5-v5rds" Oct 30 22:13:30.748682 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:30.748632 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b03b6525-1963-4875-91d9-025d1e884994-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88af5db5-v5rds\" (UID: \"b03b6525-1963-4875-91d9-025d1e884994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af5db5-v5rds" Oct 30 22:13:30.816384 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:30.816335 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:13:30.851075 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:30.851034 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b03b6525-1963-4875-91d9-025d1e884994-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88af5db5-v5rds\" (UID: \"b03b6525-1963-4875-91d9-025d1e884994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af5db5-v5rds" Oct 30 22:13:30.851350 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:30.851325 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b03b6525-1963-4875-91d9-025d1e884994-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88af5db5-v5rds\" (UID: \"b03b6525-1963-4875-91d9-025d1e884994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af5db5-v5rds" Oct 30 22:13:30.851544 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:30.851524 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dw5l\" (UniqueName: \"kubernetes.io/projected/b03b6525-1963-4875-91d9-025d1e884994-kube-api-access-8dw5l\") pod \"maas-api-revoke-sub-88af5db5-v5rds\" (UID: \"b03b6525-1963-4875-91d9-025d1e884994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af5db5-v5rds" Oct 30 22:13:30.852237 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:30.852210 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b03b6525-1963-4875-91d9-025d1e884994-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88af5db5-v5rds\" (UID: \"b03b6525-1963-4875-91d9-025d1e884994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af5db5-v5rds" Oct 30 22:13:30.858428 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:30.854068 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b03b6525-1963-4875-91d9-025d1e884994-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88af5db5-v5rds\" (UID: \"b03b6525-1963-4875-91d9-025d1e884994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af5db5-v5rds" Oct 30 22:13:30.922081 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:30.922045 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dw5l\" (UniqueName: \"kubernetes.io/projected/b03b6525-1963-4875-91d9-025d1e884994-kube-api-access-8dw5l\") pod \"maas-api-revoke-sub-88af5db5-v5rds\" (UID: \"b03b6525-1963-4875-91d9-025d1e884994\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af5db5-v5rds" Oct 30 22:13:30.985782 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:30.985728 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af5db5-v5rds" Oct 30 22:13:31.432525 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:13:31.417882 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03b6525_1963_4875_91d9_025d1e884994.slice/crio-7ce2bff4174c6006fa8f885310b99dc0f310fb110dac686b9443dc2e2b27be5c WatchSource:0}: Error finding container 7ce2bff4174c6006fa8f885310b99dc0f310fb110dac686b9443dc2e2b27be5c: Status 404 returned error can't find the container with id 7ce2bff4174c6006fa8f885310b99dc0f310fb110dac686b9443dc2e2b27be5c Oct 30 22:13:31.432525 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:31.417946 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88af5db5-v5rds"] Oct 30 22:13:31.525277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:31.522860 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af5db5-v5rds" event={"ID":"b03b6525-1963-4875-91d9-025d1e884994","Type":"ContainerStarted","Data":"7ce2bff4174c6006fa8f885310b99dc0f310fb110dac686b9443dc2e2b27be5c"} Oct 30 22:13:32.797418 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:32.797370 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af5db5-v5rds" event={"ID":"b03b6525-1963-4875-91d9-025d1e884994","Type":"ContainerStarted","Data":"97f58a684ec1023b33ee7a4cb96c88be853a67528d5bbe977655a93f983a3e06"} Oct 30 22:13:32.925360 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:32.923164 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af5db5-v5rds" podStartSLOduration=2.923144813 podStartE2EDuration="2.923144813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:32.920422741 +0000 UTC m=+1415.472599041" watchObservedRunningTime="2026-10-30 22:13:32.923144813 +0000 UTC m=+1415.475321114" Oct 30 22:13:33.067277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.061524 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p"] Oct 30 22:13:33.156272 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.155511 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p"] Oct 30 22:13:33.156272 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.155654 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.356335 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.338428 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-workload-certs\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.356658 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.356617 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhphr\" (UniqueName: \"kubernetes.io/projected/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-kube-api-access-hhphr\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.421035 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.420993 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-workload-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.421391 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.421366 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-podinfo\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.421595 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.421580 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-token\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.421732 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.421717 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-data\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.421853 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.421839 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-envoy\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.432281 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.421940 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istiod-ca-cert\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.433145 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.433111 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-credential-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.640275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.636273 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-podinfo\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.640275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.636403 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-token\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.640275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.636449 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-data\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.640275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.636490 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-envoy\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.640275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.636518 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istiod-ca-cert\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.640275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.636568 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-credential-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.640275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.636715 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-workload-certs\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.640275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.636752 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhphr\" (UniqueName: \"kubernetes.io/projected/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-kube-api-access-hhphr\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.640275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.636881 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-workload-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.640275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.637464 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-workload-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.653274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.641784 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istiod-ca-cert\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.653274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.642436 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-data\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.653274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.648646 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-workload-certs\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.653274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.648924 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-credential-socket\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.773276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.768664 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhphr\" (UniqueName: \"kubernetes.io/projected/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-kube-api-access-hhphr\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.814973 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.814931 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-podinfo\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.817301 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.817272 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-token\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.850335 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.829633 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-envoy\") pod \"e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:33.899958 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.899904 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:33.995148 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:33.995089 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 22:13:34.008302 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:34.008223 6006 generic.go:356] "Generic (PLEG): container finished" podID="b03b6525-1963-4875-91d9-025d1e884994" containerID="97f58a684ec1023b33ee7a4cb96c88be853a67528d5bbe977655a93f983a3e06" exitCode=0 Oct 30 22:13:34.008616 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:34.008587 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af5db5-v5rds" event={"ID":"b03b6525-1963-4875-91d9-025d1e884994","Type":"ContainerDied","Data":"97f58a684ec1023b33ee7a4cb96c88be853a67528d5bbe977655a93f983a3e06"} Oct 30 22:13:34.083097 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:34.083060 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:34.246276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:34.245638 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 22:13:34.267434 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:34.260389 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 22:13:34.455362 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:34.439844 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:34.470317 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:34.466551 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:34.682370 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:34.419239 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:34.688445 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:34.684882 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 22:13:34.701425 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:34.697661 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 22:13:34.745285 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:34.745143 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 22:13:35.014472 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:35.009559 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 22:13:35.044356 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:35.042384 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:13:35.105394 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:35.105340 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p"] Oct 30 22:13:35.157110 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:13:35.157071 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d7d6f3_7301_4ff6_a8a9_cd2917c8cfb3.slice/crio-496df27568e514d6f03ac0ba1694637ff1e0ad4ec1acd5c626202e691bcef762 WatchSource:0}: Error finding container 496df27568e514d6f03ac0ba1694637ff1e0ad4ec1acd5c626202e691bcef762: Status 404 returned error can't find the container with id 496df27568e514d6f03ac0ba1694637ff1e0ad4ec1acd5c626202e691bcef762 Oct 30 22:13:35.183922 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:35.183863 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:13:35.184110 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:35.183979 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:13:35.184110 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:35.184031 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:13:35.213708 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:35.213512 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:13:35.247311 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:35.246657 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 22:13:35.247311 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:35.247068 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:13:35.346350 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:35.346316 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:13:35.669642 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:35.669609 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af5db5-v5rds" Oct 30 22:13:35.859703 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:35.859609 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b03b6525-1963-4875-91d9-025d1e884994-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b03b6525-1963-4875-91d9-025d1e884994-maas-api-xks-ca\") pod \"b03b6525-1963-4875-91d9-025d1e884994\" (UID: \"b03b6525-1963-4875-91d9-025d1e884994\") " Oct 30 22:13:35.859880 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:35.859714 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b03b6525-1963-4875-91d9-025d1e884994-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b03b6525-1963-4875-91d9-025d1e884994-maas-api-openshift-ca\") pod \"b03b6525-1963-4875-91d9-025d1e884994\" (UID: \"b03b6525-1963-4875-91d9-025d1e884994\") " Oct 30 22:13:35.859880 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:35.859799 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b03b6525-1963-4875-91d9-025d1e884994-kube-api-access-8dw5l\" (UniqueName: \"kubernetes.io/projected/b03b6525-1963-4875-91d9-025d1e884994-kube-api-access-8dw5l\") pod \"b03b6525-1963-4875-91d9-025d1e884994\" (UID: \"b03b6525-1963-4875-91d9-025d1e884994\") " Oct 30 22:13:35.860173 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:35.860139 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b03b6525-1963-4875-91d9-025d1e884994-maas-api-openshift-ca" pod "b03b6525-1963-4875-91d9-025d1e884994" (UID: "b03b6525-1963-4875-91d9-025d1e884994"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:35.860548 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:35.860528 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b03b6525-1963-4875-91d9-025d1e884994-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:35.862478 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:35.862448 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b03b6525-1963-4875-91d9-025d1e884994-maas-api-xks-ca" pod "b03b6525-1963-4875-91d9-025d1e884994" (UID: "b03b6525-1963-4875-91d9-025d1e884994"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:13:35.862572 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:35.862476 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03b6525-1963-4875-91d9-025d1e884994-kube-api-access-8dw5l" pod "b03b6525-1963-4875-91d9-025d1e884994" (UID: "b03b6525-1963-4875-91d9-025d1e884994"). InnerVolumeSpecName "kube-api-access-8dw5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:35.962320 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:35.962239 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dw5l\" (UniqueName: \"kubernetes.io/projected/b03b6525-1963-4875-91d9-025d1e884994-kube-api-access-8dw5l\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:35.962320 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:35.962298 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b03b6525-1963-4875-91d9-025d1e884994-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:36.046364 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:36.046321 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" event={"ID":"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3","Type":"ContainerStarted","Data":"009be692572310e3168485877a4133d7ffcab468040cfda7549f62534e356956"} Oct 30 22:13:36.046862 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:36.046395 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" event={"ID":"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3","Type":"ContainerStarted","Data":"496df27568e514d6f03ac0ba1694637ff1e0ad4ec1acd5c626202e691bcef762"} Oct 30 22:13:36.047998 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:36.047966 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af5db5-v5rds" Oct 30 22:13:36.048142 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:36.048007 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af5db5-v5rds" event={"ID":"b03b6525-1963-4875-91d9-025d1e884994","Type":"ContainerDied","Data":"7ce2bff4174c6006fa8f885310b99dc0f310fb110dac686b9443dc2e2b27be5c"} Oct 30 22:13:36.048142 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:36.048048 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ce2bff4174c6006fa8f885310b99dc0f310fb110dac686b9443dc2e2b27be5c" Oct 30 22:13:36.083999 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:36.083966 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:36.085558 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:36.085529 6006 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:13:36.085674 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:36.085580 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" podUID="f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 22:13:36.170708 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:36.170591 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" podStartSLOduration=4.170575931 podStartE2EDuration="4.170575931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:36.16373458 +0000 UTC m=+1418.715910880" watchObservedRunningTime="2026-10-30 22:13:36.170575931 +0000 UTC m=+1418.722752230" Oct 30 22:13:37.085016 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:37.084974 6006 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:13:37.085562 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:37.085032 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" podUID="f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 22:13:38.084665 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:38.084626 6006 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:13:38.084848 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:38.084695 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" podUID="f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 22:13:39.084515 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:39.084473 6006 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:13:39.084938 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:39.084533 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" podUID="f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 22:13:40.084813 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:40.084779 6006 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:13:40.085306 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:40.084849 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" podUID="f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 22:13:41.084114 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:41.084071 6006 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:13:41.084348 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:41.084139 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" podUID="f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 22:13:42.084771 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:42.084732 6006 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:13:42.085321 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:42.085284 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" podUID="f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 22:13:43.089551 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:43.089517 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:43.090029 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:43.089857 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:43.090709 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:43.090685 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:13:47.338536 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.338487 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm"] Oct 30 22:13:47.339549 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.339523 6006 cpu_manager.go:414] "containerMap: removing container" podUID="b03b6525-1963-4875-91d9-025d1e884994" containerName="revoke-keys" Oct 30 22:13:47.339549 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.339550 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b03b6525-1963-4875-91d9-025d1e884994" containerName="revoke-keys" Oct 30 22:13:47.339725 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.339717 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="b03b6525-1963-4875-91d9-025d1e884994" containerName="revoke-keys" Oct 30 22:13:47.344018 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.343993 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.445441 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.445406 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm"] Oct 30 22:13:47.475836 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.475805 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-data\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.476033 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.475854 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-workload-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.476107 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.476059 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istiod-ca-cert\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.476153 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.476128 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-workload-certs\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.476297 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.476230 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxr85\" (UniqueName: \"kubernetes.io/projected/7cce6f74-3f9a-49cf-a594-211f3bb1f432-kube-api-access-bxr85\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.476342 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.476332 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-envoy\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.476481 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.476461 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-token\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.476530 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.476518 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-podinfo\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.476563 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.476547 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-credential-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.578122 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.578086 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istiod-ca-cert\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.578122 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.578127 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-workload-certs\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.578401 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.578183 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxr85\" (UniqueName: \"kubernetes.io/projected/7cce6f74-3f9a-49cf-a594-211f3bb1f432-kube-api-access-bxr85\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.578401 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.578214 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-envoy\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.578509 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.578452 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-token\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.578561 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.578510 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-podinfo\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.578561 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.578553 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-credential-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.578650 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.578631 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-workload-certs\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.578702 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.578652 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-data\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.578757 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.578714 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-workload-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.578978 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.578951 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-data\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.579076 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.578986 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-credential-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.579076 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.578994 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istiod-ca-cert\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.579175 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.579134 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-workload-socket\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.580686 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.580665 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-envoy\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.581050 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.581028 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-podinfo\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.648512 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.648427 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5"] Oct 30 22:13:47.649745 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.649712 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxr85\" (UniqueName: \"kubernetes.io/projected/7cce6f74-3f9a-49cf-a594-211f3bb1f432-kube-api-access-bxr85\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.652537 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.652506 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-token\") pod \"e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.659900 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.659858 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:47.663747 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.663723 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.731470 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.714955 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-c9dbe6-openshift-default-dockercfg-bpxxl\"" Oct 30 22:13:47.731470 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.719517 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5"] Oct 30 22:13:47.781684 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.781642 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istiod-ca-cert\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.781870 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.781771 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-workload-certs\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.781870 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.781831 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4qwc\" (UniqueName: \"kubernetes.io/projected/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-kube-api-access-b4qwc\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.781982 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.781876 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-workload-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.781982 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.781953 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-envoy\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.782097 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.782081 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-token\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.782157 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.782138 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-credential-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.782210 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.782167 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-data\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.782280 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.782221 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-podinfo\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.883209 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.883173 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-token\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.883411 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.883243 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-credential-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.883411 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.883318 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-data\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.883411 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.883370 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-podinfo\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.883577 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.883524 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istiod-ca-cert\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.883650 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.883584 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-workload-certs\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.883650 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.883641 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4qwc\" (UniqueName: \"kubernetes.io/projected/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-kube-api-access-b4qwc\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.883760 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.883691 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-workload-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.883760 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.883725 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-credential-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.883760 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.883747 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-envoy\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.883966 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.883795 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-data\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.884039 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.883959 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-workload-certs\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.884205 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.884184 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-workload-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.884285 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.884236 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istiod-ca-cert\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.885882 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.885858 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-envoy\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.885976 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.885935 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-podinfo\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.942965 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.942845 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-token\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:47.942965 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:47.942878 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4qwc\" (UniqueName: \"kubernetes.io/projected/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-kube-api-access-b4qwc\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:48.028557 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:48.028509 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:48.116744 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:48.116704 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm"] Oct 30 22:13:48.133901 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:48.133003 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:13:48.133901 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:48.133107 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:13:48.133901 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:48.133154 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:13:48.159435 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:48.159384 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" event={"ID":"7cce6f74-3f9a-49cf-a594-211f3bb1f432","Type":"ContainerStarted","Data":"ccd212ab3a9de1535295f753211d5ec3c510648757ce5fb5314d97133fdb7190"} Oct 30 22:13:48.420301 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:48.420234 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5"] Oct 30 22:13:48.422153 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:13:48.422118 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500679e9_0b27_49c3_9b36_7fcfb7d1fc32.slice/crio-9abb7ab6bf87281395b38f305530e0c66c4bb2b70e33bcb7f8080139999aad49 WatchSource:0}: Error finding container 9abb7ab6bf87281395b38f305530e0c66c4bb2b70e33bcb7f8080139999aad49: Status 404 returned error can't find the container with id 9abb7ab6bf87281395b38f305530e0c66c4bb2b70e33bcb7f8080139999aad49 Oct 30 22:13:48.435279 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:48.429676 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:13:48.435279 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:48.429774 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:13:48.435279 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:48.429816 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:13:49.174131 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:49.174088 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" event={"ID":"7cce6f74-3f9a-49cf-a594-211f3bb1f432","Type":"ContainerStarted","Data":"69e726c5872b7976fdc0fef0721a30048e24a2ddef9f040cd58341f3e86d8fb8"} Oct 30 22:13:49.176715 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:49.176679 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" event={"ID":"500679e9-0b27-49c3-9b36-7fcfb7d1fc32","Type":"ContainerStarted","Data":"6d061c26c259af7af648cb0611c86554a942bdc9b7389bc149b075371098778e"} Oct 30 22:13:49.176868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:49.176744 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" event={"ID":"500679e9-0b27-49c3-9b36-7fcfb7d1fc32","Type":"ContainerStarted","Data":"9abb7ab6bf87281395b38f305530e0c66c4bb2b70e33bcb7f8080139999aad49"} Oct 30 22:13:49.366381 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:49.363215 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" podStartSLOduration=2.36318949 podStartE2EDuration="2.36318949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:49.350575618 +0000 UTC m=+1431.902751919" watchObservedRunningTime="2026-10-30 22:13:49.36318949 +0000 UTC m=+1431.915365790" Oct 30 22:13:49.478674 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:49.478557 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" podStartSLOduration=2.478544439 podStartE2EDuration="2.478544439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:49.477662198 +0000 UTC m=+1432.029838493" watchObservedRunningTime="2026-10-30 22:13:49.478544439 +0000 UTC m=+1432.030720739" Oct 30 22:13:49.662178 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:49.662140 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:13:49.662484 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:49.662452 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" podUID="7cce6f74-3f9a-49cf-a594-211f3bb1f432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 22:13:49.665328 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:49.665228 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:50.057274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:50.032160 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:50.057274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:50.040378 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:13:50.057274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:50.040444 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" podUID="500679e9-0b27-49c3-9b36-7fcfb7d1fc32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 22:13:50.068330 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:50.068289 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9a3e41c-tmgdw"] Oct 30 22:13:50.073571 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:50.073536 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9a3e41c-tmgdw"] Oct 30 22:13:50.660704 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:50.660668 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:13:50.661189 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:50.661162 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" podUID="7cce6f74-3f9a-49cf-a594-211f3bb1f432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 22:13:51.060346 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:51.043376 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:13:51.060346 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:51.043441 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" podUID="500679e9-0b27-49c3-9b36-7fcfb7d1fc32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 22:13:51.652297 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:51.652236 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f11701d-ecc8-4627-bb0e-6e99a8efebea" path="/var/lib/kubelet/pods/7f11701d-ecc8-4627-bb0e-6e99a8efebea/volumes" Oct 30 22:13:51.667300 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:51.665631 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:13:51.667300 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:51.665735 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" podUID="7cce6f74-3f9a-49cf-a594-211f3bb1f432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 22:13:52.042337 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:52.031703 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:13:52.042337 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:52.031806 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" podUID="500679e9-0b27-49c3-9b36-7fcfb7d1fc32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 22:13:52.665276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:52.662354 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:13:52.665276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:52.662419 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" podUID="7cce6f74-3f9a-49cf-a594-211f3bb1f432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 22:13:53.029768 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:53.029671 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:13:53.030410 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:53.030369 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" podUID="500679e9-0b27-49c3-9b36-7fcfb7d1fc32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 22:13:53.662597 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:53.662557 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:13:53.664911 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:53.664874 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" podUID="7cce6f74-3f9a-49cf-a594-211f3bb1f432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 22:13:54.035331 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:54.034663 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:13:54.035331 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:54.034729 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" podUID="500679e9-0b27-49c3-9b36-7fcfb7d1fc32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 22:13:54.058276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:54.057059 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-837c347f-q62v4"] Oct 30 22:13:54.064273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:54.063735 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-837c347f-q62v4"] Oct 30 22:13:54.660422 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:54.660388 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:13:54.660745 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:54.660612 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" podUID="7cce6f74-3f9a-49cf-a594-211f3bb1f432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 22:13:55.029651 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:55.029556 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:13:55.029651 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:55.029632 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" podUID="500679e9-0b27-49c3-9b36-7fcfb7d1fc32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 22:13:55.627293 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:55.626663 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7587b568-cf56-45a8-97f7-52d98edf7b17" path="/var/lib/kubelet/pods/7587b568-cf56-45a8-97f7-52d98edf7b17/volumes" Oct 30 22:13:56.086747 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:56.086710 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:56.087730 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:56.087705 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:56.089203 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:56.089178 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:13:56.493954 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:56.493873 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:56.494667 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:56.494631 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:56.495991 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:56.495964 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:13:57.812455 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:57.812416 6006 scope.go:122] "RemoveContainer" containerID="8fc1bf6808144cc21645e271600cc70f1982eb3785cb142e298bc0fd8f0fb5a7" Oct 30 22:13:58.059584 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:13:58.059537 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:13:59.092573 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:59.092529 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9291b53a-gdbsm"] Oct 30 22:13:59.115788 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:59.115735 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9291b53a-gdbsm"] Oct 30 22:13:59.830290 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:59.827724 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="023471a9-eb6d-410a-8fc9-4ee320f6f481" path="/var/lib/kubelet/pods/023471a9-eb6d-410a-8fc9-4ee320f6f481/volumes" Oct 30 22:13:59.935276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:13:59.934474 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:00.222466 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:00.222345 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:00.246398 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:00.246353 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 22:14:00.354413 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:00.354357 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:00.483332 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:00.482361 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:00.506406 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:00.506356 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:00.556408 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:00.556359 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 22:14:00.564295 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:00.564243 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:00.597546 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:00.597475 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:14:00.656580 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:00.656526 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:00.673381 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:00.673335 6006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:00.938159 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:00.938107 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:00.946036 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:00.946005 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:14:00.991562 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:00.991527 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:14:02.098309 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:02.098243 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bfd8ad6-4zqlc"] Oct 30 22:14:02.104241 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:02.104210 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfd8ad6-4zqlc" Oct 30 22:14:02.157367 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:02.154368 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:14:02.157367 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:02.154700 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:14:02.161437 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:02.158952 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bfd8ad6-4zqlc"] Oct 30 22:14:02.265046 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:02.265011 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775d1260-3327-48a0-a7f6-47b62265e310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bfd8ad6-4zqlc\" (UID: \"775d1260-3327-48a0-a7f6-47b62265e310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfd8ad6-4zqlc" Oct 30 22:14:02.265245 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:02.265054 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775d1260-3327-48a0-a7f6-47b62265e310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bfd8ad6-4zqlc\" (UID: \"775d1260-3327-48a0-a7f6-47b62265e310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfd8ad6-4zqlc" Oct 30 22:14:02.265245 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:02.265077 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ptww\" (UniqueName: \"kubernetes.io/projected/775d1260-3327-48a0-a7f6-47b62265e310-kube-api-access-5ptww\") pod \"maas-api-revoke-sub-8bfd8ad6-4zqlc\" (UID: \"775d1260-3327-48a0-a7f6-47b62265e310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfd8ad6-4zqlc" Oct 30 22:14:02.366331 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:02.366221 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775d1260-3327-48a0-a7f6-47b62265e310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bfd8ad6-4zqlc\" (UID: \"775d1260-3327-48a0-a7f6-47b62265e310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfd8ad6-4zqlc" Oct 30 22:14:02.366331 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:02.366289 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775d1260-3327-48a0-a7f6-47b62265e310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bfd8ad6-4zqlc\" (UID: \"775d1260-3327-48a0-a7f6-47b62265e310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfd8ad6-4zqlc" Oct 30 22:14:02.366331 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:02.366324 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ptww\" (UniqueName: \"kubernetes.io/projected/775d1260-3327-48a0-a7f6-47b62265e310-kube-api-access-5ptww\") pod \"maas-api-revoke-sub-8bfd8ad6-4zqlc\" (UID: \"775d1260-3327-48a0-a7f6-47b62265e310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfd8ad6-4zqlc" Oct 30 22:14:02.367122 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:02.367094 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775d1260-3327-48a0-a7f6-47b62265e310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bfd8ad6-4zqlc\" (UID: \"775d1260-3327-48a0-a7f6-47b62265e310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfd8ad6-4zqlc" Oct 30 22:14:02.368822 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:02.368796 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775d1260-3327-48a0-a7f6-47b62265e310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bfd8ad6-4zqlc\" (UID: \"775d1260-3327-48a0-a7f6-47b62265e310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfd8ad6-4zqlc" Oct 30 22:14:02.425621 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:02.425582 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ptww\" (UniqueName: \"kubernetes.io/projected/775d1260-3327-48a0-a7f6-47b62265e310-kube-api-access-5ptww\") pod \"maas-api-revoke-sub-8bfd8ad6-4zqlc\" (UID: \"775d1260-3327-48a0-a7f6-47b62265e310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfd8ad6-4zqlc" Oct 30 22:14:02.426191 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:02.426169 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfd8ad6-4zqlc" Oct 30 22:14:02.837666 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:02.837632 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bfd8ad6-4zqlc"] Oct 30 22:14:02.844946 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:14:02.840269 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775d1260_3327_48a0_a7f6_47b62265e310.slice/crio-74dbb9f413bb129019769bdfad289d0ec33d0a5fe252dab13ffd7da8721a0ed6 WatchSource:0}: Error finding container 74dbb9f413bb129019769bdfad289d0ec33d0a5fe252dab13ffd7da8721a0ed6: Status 404 returned error can't find the container with id 74dbb9f413bb129019769bdfad289d0ec33d0a5fe252dab13ffd7da8721a0ed6 Oct 30 22:14:03.839851 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:03.839814 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfd8ad6-4zqlc" event={"ID":"775d1260-3327-48a0-a7f6-47b62265e310","Type":"ContainerStarted","Data":"a4ab22e2ab892ab5f7b26a7d02d39557a9bfffba9608422bb89fae132b28e721"} Oct 30 22:14:03.840401 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:03.839883 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfd8ad6-4zqlc" event={"ID":"775d1260-3327-48a0-a7f6-47b62265e310","Type":"ContainerStarted","Data":"74dbb9f413bb129019769bdfad289d0ec33d0a5fe252dab13ffd7da8721a0ed6"} Oct 30 22:14:03.953350 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:03.953294 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfd8ad6-4zqlc" podStartSLOduration=1.953279483 podStartE2EDuration="1.953279483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:03.950665115 +0000 UTC m=+1446.502841418" watchObservedRunningTime="2026-10-30 22:14:03.953279483 +0000 UTC m=+1446.505455783" Oct 30 22:14:04.853406 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:04.853368 6006 generic.go:356] "Generic (PLEG): container finished" podID="775d1260-3327-48a0-a7f6-47b62265e310" containerID="a4ab22e2ab892ab5f7b26a7d02d39557a9bfffba9608422bb89fae132b28e721" exitCode=0 Oct 30 22:14:04.853794 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:04.853449 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfd8ad6-4zqlc" event={"ID":"775d1260-3327-48a0-a7f6-47b62265e310","Type":"ContainerDied","Data":"a4ab22e2ab892ab5f7b26a7d02d39557a9bfffba9608422bb89fae132b28e721"} Oct 30 22:14:06.040648 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:06.040614 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfd8ad6-4zqlc" Oct 30 22:14:06.066642 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:06.066614 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c65ead47-qssdz"] Oct 30 22:14:06.113363 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:06.113275 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c65ead47-qssdz"] Oct 30 22:14:06.129729 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:06.129697 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/775d1260-3327-48a0-a7f6-47b62265e310-kube-api-access-5ptww\" (UniqueName: \"kubernetes.io/projected/775d1260-3327-48a0-a7f6-47b62265e310-kube-api-access-5ptww\") pod \"775d1260-3327-48a0-a7f6-47b62265e310\" (UID: \"775d1260-3327-48a0-a7f6-47b62265e310\") " Oct 30 22:14:06.129898 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:06.129780 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/775d1260-3327-48a0-a7f6-47b62265e310-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775d1260-3327-48a0-a7f6-47b62265e310-maas-api-xks-ca\") pod \"775d1260-3327-48a0-a7f6-47b62265e310\" (UID: \"775d1260-3327-48a0-a7f6-47b62265e310\") " Oct 30 22:14:06.130064 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:06.130048 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/775d1260-3327-48a0-a7f6-47b62265e310-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775d1260-3327-48a0-a7f6-47b62265e310-maas-api-openshift-ca\") pod \"775d1260-3327-48a0-a7f6-47b62265e310\" (UID: \"775d1260-3327-48a0-a7f6-47b62265e310\") " Oct 30 22:14:06.130499 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:06.130462 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775d1260-3327-48a0-a7f6-47b62265e310-maas-api-openshift-ca" pod "775d1260-3327-48a0-a7f6-47b62265e310" (UID: "775d1260-3327-48a0-a7f6-47b62265e310"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:06.130859 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:06.130826 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775d1260-3327-48a0-a7f6-47b62265e310-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:06.132223 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:06.132189 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775d1260-3327-48a0-a7f6-47b62265e310-kube-api-access-5ptww" pod "775d1260-3327-48a0-a7f6-47b62265e310" (UID: "775d1260-3327-48a0-a7f6-47b62265e310"). InnerVolumeSpecName "kube-api-access-5ptww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:06.132462 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:06.132434 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/775d1260-3327-48a0-a7f6-47b62265e310-maas-api-xks-ca" pod "775d1260-3327-48a0-a7f6-47b62265e310" (UID: "775d1260-3327-48a0-a7f6-47b62265e310"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:06.240131 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:06.240087 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ptww\" (UniqueName: \"kubernetes.io/projected/775d1260-3327-48a0-a7f6-47b62265e310-kube-api-access-5ptww\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:06.240406 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:06.240386 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775d1260-3327-48a0-a7f6-47b62265e310-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:06.869685 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:06.869648 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfd8ad6-4zqlc" Oct 30 22:14:06.869861 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:06.869655 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfd8ad6-4zqlc" event={"ID":"775d1260-3327-48a0-a7f6-47b62265e310","Type":"ContainerDied","Data":"74dbb9f413bb129019769bdfad289d0ec33d0a5fe252dab13ffd7da8721a0ed6"} Oct 30 22:14:06.869861 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:06.869772 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74dbb9f413bb129019769bdfad289d0ec33d0a5fe252dab13ffd7da8721a0ed6" Oct 30 22:14:07.607039 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:07.607001 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f71ebd22-a1d9-4f0c-b56c-8a98f7908255" path="/var/lib/kubelet/pods/f71ebd22-a1d9-4f0c-b56c-8a98f7908255/volumes" Oct 30 22:14:12.420510 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:12.420470 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-702f477e-xtz54"] Oct 30 22:14:12.421524 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:12.421502 6006 cpu_manager.go:414] "containerMap: removing container" podUID="775d1260-3327-48a0-a7f6-47b62265e310" containerName="revoke-keys" Oct 30 22:14:12.421587 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:12.421527 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="775d1260-3327-48a0-a7f6-47b62265e310" containerName="revoke-keys" Oct 30 22:14:12.421772 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:12.421759 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="775d1260-3327-48a0-a7f6-47b62265e310" containerName="revoke-keys" Oct 30 22:14:12.427346 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:12.427321 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-702f477e-xtz54" Oct 30 22:14:12.478029 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:12.477993 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-702f477e-xtz54"] Oct 30 22:14:12.519920 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:12.519888 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:14:12.519920 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:12.519888 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:14:12.550333 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:12.550295 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12332804-45e1-4613-b6ed-1e0f7ca91b65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-702f477e-xtz54\" (UID: \"12332804-45e1-4613-b6ed-1e0f7ca91b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-702f477e-xtz54" Oct 30 22:14:12.550543 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:12.550459 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12332804-45e1-4613-b6ed-1e0f7ca91b65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-702f477e-xtz54\" (UID: \"12332804-45e1-4613-b6ed-1e0f7ca91b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-702f477e-xtz54" Oct 30 22:14:12.550678 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:12.550645 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lckc7\" (UniqueName: \"kubernetes.io/projected/12332804-45e1-4613-b6ed-1e0f7ca91b65-kube-api-access-lckc7\") pod \"maas-api-revoke-sub-702f477e-xtz54\" (UID: \"12332804-45e1-4613-b6ed-1e0f7ca91b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-702f477e-xtz54" Oct 30 22:14:12.652906 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:12.652860 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12332804-45e1-4613-b6ed-1e0f7ca91b65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-702f477e-xtz54\" (UID: \"12332804-45e1-4613-b6ed-1e0f7ca91b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-702f477e-xtz54" Oct 30 22:14:12.653094 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:12.653014 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lckc7\" (UniqueName: \"kubernetes.io/projected/12332804-45e1-4613-b6ed-1e0f7ca91b65-kube-api-access-lckc7\") pod \"maas-api-revoke-sub-702f477e-xtz54\" (UID: \"12332804-45e1-4613-b6ed-1e0f7ca91b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-702f477e-xtz54" Oct 30 22:14:12.653326 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:12.653263 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12332804-45e1-4613-b6ed-1e0f7ca91b65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-702f477e-xtz54\" (UID: \"12332804-45e1-4613-b6ed-1e0f7ca91b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-702f477e-xtz54" Oct 30 22:14:12.653997 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:12.653971 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12332804-45e1-4613-b6ed-1e0f7ca91b65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-702f477e-xtz54\" (UID: \"12332804-45e1-4613-b6ed-1e0f7ca91b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-702f477e-xtz54" Oct 30 22:14:12.655447 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:12.655417 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12332804-45e1-4613-b6ed-1e0f7ca91b65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-702f477e-xtz54\" (UID: \"12332804-45e1-4613-b6ed-1e0f7ca91b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-702f477e-xtz54" Oct 30 22:14:12.711733 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:12.711644 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lckc7\" (UniqueName: \"kubernetes.io/projected/12332804-45e1-4613-b6ed-1e0f7ca91b65-kube-api-access-lckc7\") pod \"maas-api-revoke-sub-702f477e-xtz54\" (UID: \"12332804-45e1-4613-b6ed-1e0f7ca91b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-702f477e-xtz54" Oct 30 22:14:12.744989 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:12.744958 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-702f477e-xtz54" Oct 30 22:14:13.125548 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:14:13.125516 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12332804_45e1_4613_b6ed_1e0f7ca91b65.slice/crio-ab093bdfc820c4b86be021a280d96d3fc4751cf215014affc4a6e4bdc153c4cc WatchSource:0}: Error finding container ab093bdfc820c4b86be021a280d96d3fc4751cf215014affc4a6e4bdc153c4cc: Status 404 returned error can't find the container with id ab093bdfc820c4b86be021a280d96d3fc4751cf215014affc4a6e4bdc153c4cc Oct 30 22:14:13.126008 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:13.125936 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-702f477e-xtz54"] Oct 30 22:14:13.931161 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:13.931124 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-702f477e-xtz54" event={"ID":"12332804-45e1-4613-b6ed-1e0f7ca91b65","Type":"ContainerStarted","Data":"453f616f3eecb2d406d359b37ab843ccceded783d6881bb12d4a7f51c594f0f8"} Oct 30 22:14:13.931611 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:13.931185 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-702f477e-xtz54" event={"ID":"12332804-45e1-4613-b6ed-1e0f7ca91b65","Type":"ContainerStarted","Data":"ab093bdfc820c4b86be021a280d96d3fc4751cf215014affc4a6e4bdc153c4cc"} Oct 30 22:14:14.044548 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:14.044495 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-702f477e-xtz54" podStartSLOduration=2.044479828 podStartE2EDuration="2.044479828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:14.042027742 +0000 UTC m=+1456.594204042" watchObservedRunningTime="2026-10-30 22:14:14.044479828 +0000 UTC m=+1456.596656127" Oct 30 22:14:14.940427 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:14.940390 6006 generic.go:356] "Generic (PLEG): container finished" podID="12332804-45e1-4613-b6ed-1e0f7ca91b65" containerID="453f616f3eecb2d406d359b37ab843ccceded783d6881bb12d4a7f51c594f0f8" exitCode=0 Oct 30 22:14:14.940849 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:14.940470 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-702f477e-xtz54" event={"ID":"12332804-45e1-4613-b6ed-1e0f7ca91b65","Type":"ContainerDied","Data":"453f616f3eecb2d406d359b37ab843ccceded783d6881bb12d4a7f51c594f0f8"} Oct 30 22:14:16.146588 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:16.146561 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-702f477e-xtz54" Oct 30 22:14:16.341761 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:16.341727 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12332804-45e1-4613-b6ed-1e0f7ca91b65-kube-api-access-lckc7\" (UniqueName: \"kubernetes.io/projected/12332804-45e1-4613-b6ed-1e0f7ca91b65-kube-api-access-lckc7\") pod \"12332804-45e1-4613-b6ed-1e0f7ca91b65\" (UID: \"12332804-45e1-4613-b6ed-1e0f7ca91b65\") " Oct 30 22:14:16.341985 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:16.341946 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12332804-45e1-4613-b6ed-1e0f7ca91b65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12332804-45e1-4613-b6ed-1e0f7ca91b65-maas-api-openshift-ca\") pod \"12332804-45e1-4613-b6ed-1e0f7ca91b65\" (UID: \"12332804-45e1-4613-b6ed-1e0f7ca91b65\") " Oct 30 22:14:16.342117 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:16.341986 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12332804-45e1-4613-b6ed-1e0f7ca91b65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12332804-45e1-4613-b6ed-1e0f7ca91b65-maas-api-xks-ca\") pod \"12332804-45e1-4613-b6ed-1e0f7ca91b65\" (UID: \"12332804-45e1-4613-b6ed-1e0f7ca91b65\") " Oct 30 22:14:16.342323 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:16.342297 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12332804-45e1-4613-b6ed-1e0f7ca91b65-maas-api-openshift-ca" pod "12332804-45e1-4613-b6ed-1e0f7ca91b65" (UID: "12332804-45e1-4613-b6ed-1e0f7ca91b65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:16.342668 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:16.342650 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12332804-45e1-4613-b6ed-1e0f7ca91b65-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.343880 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:16.343856 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12332804-45e1-4613-b6ed-1e0f7ca91b65-kube-api-access-lckc7" pod "12332804-45e1-4613-b6ed-1e0f7ca91b65" (UID: "12332804-45e1-4613-b6ed-1e0f7ca91b65"). InnerVolumeSpecName "kube-api-access-lckc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:16.344050 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:16.344027 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12332804-45e1-4613-b6ed-1e0f7ca91b65-maas-api-xks-ca" pod "12332804-45e1-4613-b6ed-1e0f7ca91b65" (UID: "12332804-45e1-4613-b6ed-1e0f7ca91b65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:16.444732 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:16.444700 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12332804-45e1-4613-b6ed-1e0f7ca91b65-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.444732 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:16.444731 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lckc7\" (UniqueName: \"kubernetes.io/projected/12332804-45e1-4613-b6ed-1e0f7ca91b65-kube-api-access-lckc7\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.959422 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:16.959392 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-702f477e-xtz54" Oct 30 22:14:16.959609 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:16.959391 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-702f477e-xtz54" event={"ID":"12332804-45e1-4613-b6ed-1e0f7ca91b65","Type":"ContainerDied","Data":"ab093bdfc820c4b86be021a280d96d3fc4751cf215014affc4a6e4bdc153c4cc"} Oct 30 22:14:16.959609 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:16.959521 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab093bdfc820c4b86be021a280d96d3fc4751cf215014affc4a6e4bdc153c4cc" Oct 30 22:14:22.777714 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:22.777675 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9de8e643-zn27k"] Oct 30 22:14:22.778377 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:22.778362 6006 cpu_manager.go:414] "containerMap: removing container" podUID="12332804-45e1-4613-b6ed-1e0f7ca91b65" containerName="revoke-keys" Oct 30 22:14:22.778377 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:22.778378 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12332804-45e1-4613-b6ed-1e0f7ca91b65" containerName="revoke-keys" Oct 30 22:14:22.778563 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:22.778552 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="12332804-45e1-4613-b6ed-1e0f7ca91b65" containerName="revoke-keys" Oct 30 22:14:22.783372 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:22.783345 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de8e643-zn27k" Oct 30 22:14:22.838772 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:22.838727 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9de8e643-zn27k"] Oct 30 22:14:22.848921 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:22.848879 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20468fa7-f3f8-46d1-a05e-6744f4ada9e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9de8e643-zn27k\" (UID: \"20468fa7-f3f8-46d1-a05e-6744f4ada9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de8e643-zn27k" Oct 30 22:14:22.849132 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:22.849107 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20468fa7-f3f8-46d1-a05e-6744f4ada9e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9de8e643-zn27k\" (UID: \"20468fa7-f3f8-46d1-a05e-6744f4ada9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de8e643-zn27k" Oct 30 22:14:22.849235 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:22.849218 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r8tq\" (UniqueName: \"kubernetes.io/projected/20468fa7-f3f8-46d1-a05e-6744f4ada9e6-kube-api-access-6r8tq\") pod \"maas-api-revoke-sub-9de8e643-zn27k\" (UID: \"20468fa7-f3f8-46d1-a05e-6744f4ada9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de8e643-zn27k" Oct 30 22:14:22.879686 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:22.879645 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:14:22.879892 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:22.879687 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:14:22.950228 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:22.950188 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r8tq\" (UniqueName: \"kubernetes.io/projected/20468fa7-f3f8-46d1-a05e-6744f4ada9e6-kube-api-access-6r8tq\") pod \"maas-api-revoke-sub-9de8e643-zn27k\" (UID: \"20468fa7-f3f8-46d1-a05e-6744f4ada9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de8e643-zn27k" Oct 30 22:14:22.950461 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:22.950361 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20468fa7-f3f8-46d1-a05e-6744f4ada9e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9de8e643-zn27k\" (UID: \"20468fa7-f3f8-46d1-a05e-6744f4ada9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de8e643-zn27k" Oct 30 22:14:22.950516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:22.950464 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20468fa7-f3f8-46d1-a05e-6744f4ada9e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9de8e643-zn27k\" (UID: \"20468fa7-f3f8-46d1-a05e-6744f4ada9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de8e643-zn27k" Oct 30 22:14:22.951018 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:22.950986 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20468fa7-f3f8-46d1-a05e-6744f4ada9e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9de8e643-zn27k\" (UID: \"20468fa7-f3f8-46d1-a05e-6744f4ada9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de8e643-zn27k" Oct 30 22:14:22.952989 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:22.952967 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20468fa7-f3f8-46d1-a05e-6744f4ada9e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9de8e643-zn27k\" (UID: \"20468fa7-f3f8-46d1-a05e-6744f4ada9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de8e643-zn27k" Oct 30 22:14:23.008898 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:23.008862 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r8tq\" (UniqueName: \"kubernetes.io/projected/20468fa7-f3f8-46d1-a05e-6744f4ada9e6-kube-api-access-6r8tq\") pod \"maas-api-revoke-sub-9de8e643-zn27k\" (UID: \"20468fa7-f3f8-46d1-a05e-6744f4ada9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de8e643-zn27k" Oct 30 22:14:23.097847 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:23.097811 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de8e643-zn27k" Oct 30 22:14:23.492832 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:23.492802 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9de8e643-zn27k"] Oct 30 22:14:23.494111 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:14:23.494085 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20468fa7_f3f8_46d1_a05e_6744f4ada9e6.slice/crio-fef12b5e6534e030803339c5261e8baa39d4b3144a38064b75d4452cbee5a740 WatchSource:0}: Error finding container fef12b5e6534e030803339c5261e8baa39d4b3144a38064b75d4452cbee5a740: Status 404 returned error can't find the container with id fef12b5e6534e030803339c5261e8baa39d4b3144a38064b75d4452cbee5a740 Oct 30 22:14:24.018017 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:24.017977 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de8e643-zn27k" event={"ID":"20468fa7-f3f8-46d1-a05e-6744f4ada9e6","Type":"ContainerStarted","Data":"372ce842e1fed14f9149049ffb270c3570069964e6f7f939d246fbf6110f8a09"} Oct 30 22:14:24.018459 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:24.018043 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de8e643-zn27k" event={"ID":"20468fa7-f3f8-46d1-a05e-6744f4ada9e6","Type":"ContainerStarted","Data":"fef12b5e6534e030803339c5261e8baa39d4b3144a38064b75d4452cbee5a740"} Oct 30 22:14:24.138964 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:24.138903 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de8e643-zn27k" podStartSLOduration=2.13888493 podStartE2EDuration="2.13888493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:24.130151331 +0000 UTC m=+1466.682327641" watchObservedRunningTime="2026-10-30 22:14:24.13888493 +0000 UTC m=+1466.691061230" Oct 30 22:14:25.039895 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:25.039123 6006 generic.go:356] "Generic (PLEG): container finished" podID="20468fa7-f3f8-46d1-a05e-6744f4ada9e6" containerID="372ce842e1fed14f9149049ffb270c3570069964e6f7f939d246fbf6110f8a09" exitCode=0 Oct 30 22:14:25.039895 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:25.039415 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de8e643-zn27k" event={"ID":"20468fa7-f3f8-46d1-a05e-6744f4ada9e6","Type":"ContainerDied","Data":"372ce842e1fed14f9149049ffb270c3570069964e6f7f939d246fbf6110f8a09"} Oct 30 22:14:26.308352 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:26.308239 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de8e643-zn27k" Oct 30 22:14:26.432334 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:26.432293 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20468fa7-f3f8-46d1-a05e-6744f4ada9e6-kube-api-access-6r8tq\" (UniqueName: \"kubernetes.io/projected/20468fa7-f3f8-46d1-a05e-6744f4ada9e6-kube-api-access-6r8tq\") pod \"20468fa7-f3f8-46d1-a05e-6744f4ada9e6\" (UID: \"20468fa7-f3f8-46d1-a05e-6744f4ada9e6\") " Oct 30 22:14:26.432538 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:26.432485 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20468fa7-f3f8-46d1-a05e-6744f4ada9e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20468fa7-f3f8-46d1-a05e-6744f4ada9e6-maas-api-openshift-ca\") pod \"20468fa7-f3f8-46d1-a05e-6744f4ada9e6\" (UID: \"20468fa7-f3f8-46d1-a05e-6744f4ada9e6\") " Oct 30 22:14:26.432659 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:26.432642 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20468fa7-f3f8-46d1-a05e-6744f4ada9e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20468fa7-f3f8-46d1-a05e-6744f4ada9e6-maas-api-xks-ca\") pod \"20468fa7-f3f8-46d1-a05e-6744f4ada9e6\" (UID: \"20468fa7-f3f8-46d1-a05e-6744f4ada9e6\") " Oct 30 22:14:26.432896 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:26.432858 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20468fa7-f3f8-46d1-a05e-6744f4ada9e6-maas-api-openshift-ca" pod "20468fa7-f3f8-46d1-a05e-6744f4ada9e6" (UID: "20468fa7-f3f8-46d1-a05e-6744f4ada9e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:26.433403 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:26.433380 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20468fa7-f3f8-46d1-a05e-6744f4ada9e6-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:26.436661 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:26.436346 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20468fa7-f3f8-46d1-a05e-6744f4ada9e6-kube-api-access-6r8tq" pod "20468fa7-f3f8-46d1-a05e-6744f4ada9e6" (UID: "20468fa7-f3f8-46d1-a05e-6744f4ada9e6"). InnerVolumeSpecName "kube-api-access-6r8tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:26.441683 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:26.441643 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20468fa7-f3f8-46d1-a05e-6744f4ada9e6-maas-api-xks-ca" pod "20468fa7-f3f8-46d1-a05e-6744f4ada9e6" (UID: "20468fa7-f3f8-46d1-a05e-6744f4ada9e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:26.535153 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:26.535110 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r8tq\" (UniqueName: \"kubernetes.io/projected/20468fa7-f3f8-46d1-a05e-6744f4ada9e6-kube-api-access-6r8tq\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:26.535153 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:26.535147 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20468fa7-f3f8-46d1-a05e-6744f4ada9e6-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:27.060854 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:27.060826 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de8e643-zn27k" Oct 30 22:14:27.061035 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:27.060864 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de8e643-zn27k" event={"ID":"20468fa7-f3f8-46d1-a05e-6744f4ada9e6","Type":"ContainerDied","Data":"fef12b5e6534e030803339c5261e8baa39d4b3144a38064b75d4452cbee5a740"} Oct 30 22:14:27.061035 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:27.060914 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fef12b5e6534e030803339c5261e8baa39d4b3144a38064b75d4452cbee5a740" Oct 30 22:14:27.354457 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:27.354371 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q"] Oct 30 22:14:27.354891 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:27.354582 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" podUID="39c47890-6d91-4569-8e7d-74114231e81c" containerName="istio-proxy" containerID="cri-o://c9a749d2e48b6e262628a8a9de7a14510583638c399e3f1571fb566d6a521776" gracePeriod=30 Oct 30 22:14:27.915183 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:14:27.915140 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:14:28.005219 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:28.005178 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a709f15-bcvzh"] Oct 30 22:14:28.006310 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:28.006287 6006 cpu_manager.go:414] "containerMap: removing container" podUID="20468fa7-f3f8-46d1-a05e-6744f4ada9e6" containerName="revoke-keys" Oct 30 22:14:28.006371 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:28.006312 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20468fa7-f3f8-46d1-a05e-6744f4ada9e6" containerName="revoke-keys" Oct 30 22:14:28.006509 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:28.006496 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="20468fa7-f3f8-46d1-a05e-6744f4ada9e6" containerName="revoke-keys" Oct 30 22:14:28.011164 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:28.011137 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a709f15-bcvzh" Oct 30 22:14:28.063174 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:28.063134 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a709f15-bcvzh"] Oct 30 22:14:28.104451 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:28.104399 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:14:28.104451 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:28.104435 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:14:28.161794 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:28.161758 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5e21fa-7968-45c9-a8d8-dc086e155f65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a709f15-bcvzh\" (UID: \"8f5e21fa-7968-45c9-a8d8-dc086e155f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a709f15-bcvzh" Oct 30 22:14:28.161983 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:28.161920 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5e21fa-7968-45c9-a8d8-dc086e155f65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a709f15-bcvzh\" (UID: \"8f5e21fa-7968-45c9-a8d8-dc086e155f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a709f15-bcvzh" Oct 30 22:14:28.162035 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:28.161979 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lglrs\" (UniqueName: \"kubernetes.io/projected/8f5e21fa-7968-45c9-a8d8-dc086e155f65-kube-api-access-lglrs\") pod \"maas-api-revoke-sub-0a709f15-bcvzh\" (UID: \"8f5e21fa-7968-45c9-a8d8-dc086e155f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a709f15-bcvzh" Oct 30 22:14:28.263339 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:28.263229 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5e21fa-7968-45c9-a8d8-dc086e155f65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a709f15-bcvzh\" (UID: \"8f5e21fa-7968-45c9-a8d8-dc086e155f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a709f15-bcvzh" Oct 30 22:14:28.263339 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:28.263333 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lglrs\" (UniqueName: \"kubernetes.io/projected/8f5e21fa-7968-45c9-a8d8-dc086e155f65-kube-api-access-lglrs\") pod \"maas-api-revoke-sub-0a709f15-bcvzh\" (UID: \"8f5e21fa-7968-45c9-a8d8-dc086e155f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a709f15-bcvzh" Oct 30 22:14:28.263604 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:28.263584 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5e21fa-7968-45c9-a8d8-dc086e155f65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a709f15-bcvzh\" (UID: \"8f5e21fa-7968-45c9-a8d8-dc086e155f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a709f15-bcvzh" Oct 30 22:14:28.264409 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:28.264382 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5e21fa-7968-45c9-a8d8-dc086e155f65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a709f15-bcvzh\" (UID: \"8f5e21fa-7968-45c9-a8d8-dc086e155f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a709f15-bcvzh" Oct 30 22:14:28.268621 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:28.268594 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5e21fa-7968-45c9-a8d8-dc086e155f65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a709f15-bcvzh\" (UID: \"8f5e21fa-7968-45c9-a8d8-dc086e155f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a709f15-bcvzh" Oct 30 22:14:28.322456 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:28.322423 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lglrs\" (UniqueName: \"kubernetes.io/projected/8f5e21fa-7968-45c9-a8d8-dc086e155f65-kube-api-access-lglrs\") pod \"maas-api-revoke-sub-0a709f15-bcvzh\" (UID: \"8f5e21fa-7968-45c9-a8d8-dc086e155f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a709f15-bcvzh" Oct 30 22:14:28.327298 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:28.327236 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a709f15-bcvzh" Oct 30 22:14:28.726915 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:28.726886 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a709f15-bcvzh"] Oct 30 22:14:28.729188 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:14:28.729154 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5e21fa_7968_45c9_a8d8_dc086e155f65.slice/crio-676d5b6df1aebed717c1df1ccabd1ac0d9f4598261ec16470fa2121d72f881d7 WatchSource:0}: Error finding container 676d5b6df1aebed717c1df1ccabd1ac0d9f4598261ec16470fa2121d72f881d7: Status 404 returned error can't find the container with id 676d5b6df1aebed717c1df1ccabd1ac0d9f4598261ec16470fa2121d72f881d7 Oct 30 22:14:29.080875 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:29.080834 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a709f15-bcvzh" event={"ID":"8f5e21fa-7968-45c9-a8d8-dc086e155f65","Type":"ContainerStarted","Data":"37c8d8e1c0aced2dda20c6ce9032ced00b5048bf2b5543cc1aa839489e313d26"} Oct 30 22:14:29.081087 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:29.080899 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a709f15-bcvzh" event={"ID":"8f5e21fa-7968-45c9-a8d8-dc086e155f65","Type":"ContainerStarted","Data":"676d5b6df1aebed717c1df1ccabd1ac0d9f4598261ec16470fa2121d72f881d7"} Oct 30 22:14:29.193782 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:29.193723 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a709f15-bcvzh" podStartSLOduration=2.193710072 podStartE2EDuration="2.193710072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:29.191691199 +0000 UTC m=+1471.743867499" watchObservedRunningTime="2026-10-30 22:14:29.193710072 +0000 UTC m=+1471.745886368" Oct 30 22:14:30.090165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:30.090130 6006 generic.go:356] "Generic (PLEG): container finished" podID="8f5e21fa-7968-45c9-a8d8-dc086e155f65" containerID="37c8d8e1c0aced2dda20c6ce9032ced00b5048bf2b5543cc1aa839489e313d26" exitCode=0 Oct 30 22:14:30.090708 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:30.090218 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a709f15-bcvzh" event={"ID":"8f5e21fa-7968-45c9-a8d8-dc086e155f65","Type":"ContainerDied","Data":"37c8d8e1c0aced2dda20c6ce9032ced00b5048bf2b5543cc1aa839489e313d26"} Oct 30 22:14:30.106269 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:30.106224 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:14:32.413276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:32.402990 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a709f15-bcvzh" Oct 30 22:14:32.539274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:32.537710 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:32.570329 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:32.558390 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:32.684274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:32.661272 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:32.684274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:32.680242 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f5e21fa-7968-45c9-a8d8-dc086e155f65-kube-api-access-lglrs\" (UniqueName: \"kubernetes.io/projected/8f5e21fa-7968-45c9-a8d8-dc086e155f65-kube-api-access-lglrs\") pod \"8f5e21fa-7968-45c9-a8d8-dc086e155f65\" (UID: \"8f5e21fa-7968-45c9-a8d8-dc086e155f65\") " Oct 30 22:14:32.684274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:32.680369 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f5e21fa-7968-45c9-a8d8-dc086e155f65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5e21fa-7968-45c9-a8d8-dc086e155f65-maas-api-xks-ca\") pod \"8f5e21fa-7968-45c9-a8d8-dc086e155f65\" (UID: \"8f5e21fa-7968-45c9-a8d8-dc086e155f65\") " Oct 30 22:14:32.684274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:32.680434 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f5e21fa-7968-45c9-a8d8-dc086e155f65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5e21fa-7968-45c9-a8d8-dc086e155f65-maas-api-openshift-ca\") pod \"8f5e21fa-7968-45c9-a8d8-dc086e155f65\" (UID: \"8f5e21fa-7968-45c9-a8d8-dc086e155f65\") " Oct 30 22:14:32.739275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:32.714733 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 22:14:32.806395 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:32.763353 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f5e21fa-7968-45c9-a8d8-dc086e155f65-maas-api-openshift-ca" pod "8f5e21fa-7968-45c9-a8d8-dc086e155f65" (UID: "8f5e21fa-7968-45c9-a8d8-dc086e155f65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:32.842328 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:32.826334 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5e21fa-7968-45c9-a8d8-dc086e155f65-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:32.981918 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:32.981839 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f5e21fa-7968-45c9-a8d8-dc086e155f65-maas-api-xks-ca" pod "8f5e21fa-7968-45c9-a8d8-dc086e155f65" (UID: "8f5e21fa-7968-45c9-a8d8-dc086e155f65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:32.999743 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:32.999685 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5e21fa-7968-45c9-a8d8-dc086e155f65-kube-api-access-lglrs" pod "8f5e21fa-7968-45c9-a8d8-dc086e155f65" (UID: "8f5e21fa-7968-45c9-a8d8-dc086e155f65"). InnerVolumeSpecName "kube-api-access-lglrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:33.053701 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:33.035652 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lglrs\" (UniqueName: \"kubernetes.io/projected/8f5e21fa-7968-45c9-a8d8-dc086e155f65-kube-api-access-lglrs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:33.053701 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:33.035694 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5e21fa-7968-45c9-a8d8-dc086e155f65-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:33.257954 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:33.239358 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:33.408276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:33.396453 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:33.436655 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:33.436608 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:33.443805 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:33.442052 6006 generic.go:356] "Generic (PLEG): container finished" podID="39c47890-6d91-4569-8e7d-74114231e81c" containerID="c9a749d2e48b6e262628a8a9de7a14510583638c399e3f1571fb566d6a521776" exitCode=0 Oct 30 22:14:33.443805 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:33.442282 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" event={"ID":"39c47890-6d91-4569-8e7d-74114231e81c","Type":"ContainerDied","Data":"c9a749d2e48b6e262628a8a9de7a14510583638c399e3f1571fb566d6a521776"} Oct 30 22:14:33.461566 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:33.461501 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a709f15-bcvzh" event={"ID":"8f5e21fa-7968-45c9-a8d8-dc086e155f65","Type":"ContainerDied","Data":"676d5b6df1aebed717c1df1ccabd1ac0d9f4598261ec16470fa2121d72f881d7"} Oct 30 22:14:33.461910 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:33.461883 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="676d5b6df1aebed717c1df1ccabd1ac0d9f4598261ec16470fa2121d72f881d7" Oct 30 22:14:33.462148 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:33.462131 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a709f15-bcvzh" Oct 30 22:14:33.530022 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:33.529983 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:33.531455 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:33.530186 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:33.546672 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:33.546619 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 22:14:33.548196 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:33.548167 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:14:33.571490 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:33.570365 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:33.573500 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:33.573472 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:14:33.669648 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:33.667596 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:33.779335 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:33.778448 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:14:33.866284 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:33.853820 6006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": context deadline exceeded" Oct 30 22:14:34.023275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:34.017931 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:34.316276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:34.308786 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 22:14:34.337301 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:34.335778 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:34.337301 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:34.336128 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:14:34.417277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:34.416528 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 22:14:34.442275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:34.417858 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:14:34.509275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:34.497529 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:14:34.509275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:34.498990 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:14:34.570368 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:34.555331 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:34.595525 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:34.572442 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:34.595525 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:34.572818 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:14:34.595525 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:34.573484 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 22:14:34.595525 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:34.573805 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:14:34.916893 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:34.617924 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:34.916893 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:34.699032 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:34.916893 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:34.699522 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:14:34.916893 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:34.716351 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:35.469548 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:35.469495 6006 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-8w8b5 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.21:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:14:35.563278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:35.518909 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-8w8b5" podUID="5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.21:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:35.563278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:35.499991 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 22:14:35.563278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:35.520009 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 22:14:35.563278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:35.521849 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 22:14:35.701431 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:35.701367 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:35.876275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:35.847366 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:14:35.894274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:35.859620 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 22:14:35.898303 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:35.863410 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:35.898303 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:35.897751 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:35.918406 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:35.912818 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:14:35.922664 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:35.922620 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:35.923470 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:35.923445 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:14:36.206468 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.192587 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 22:14:36.297278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.268772 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39c47890-6d91-4569-8e7d-74114231e81c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39c47890-6d91-4569-8e7d-74114231e81c-istio-podinfo\") pod \"39c47890-6d91-4569-8e7d-74114231e81c\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " Oct 30 22:14:36.343333 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.342567 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39c47890-6d91-4569-8e7d-74114231e81c-istio-token\" (UniqueName: \"kubernetes.io/projected/39c47890-6d91-4569-8e7d-74114231e81c-istio-token\") pod \"39c47890-6d91-4569-8e7d-74114231e81c\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " Oct 30 22:14:36.343915 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.343893 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39c47890-6d91-4569-8e7d-74114231e81c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39c47890-6d91-4569-8e7d-74114231e81c-istiod-ca-cert\") pod \"39c47890-6d91-4569-8e7d-74114231e81c\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " Oct 30 22:14:36.357506 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.357471 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-istio-data\") pod \"39c47890-6d91-4569-8e7d-74114231e81c\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " Oct 30 22:14:36.366276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.362554 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-workload-socket\") pod \"39c47890-6d91-4569-8e7d-74114231e81c\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " Oct 30 22:14:36.486885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.394457 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-istio-data" pod "39c47890-6d91-4569-8e7d-74114231e81c" (UID: "39c47890-6d91-4569-8e7d-74114231e81c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:36.486885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.416352 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:36.486885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.428955 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-workload-certs\") pod \"39c47890-6d91-4569-8e7d-74114231e81c\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " Oct 30 22:14:36.486885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.429037 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c47890-6d91-4569-8e7d-74114231e81c-istiod-ca-cert" pod "39c47890-6d91-4569-8e7d-74114231e81c" (UID: "39c47890-6d91-4569-8e7d-74114231e81c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:36.486885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.429354 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-istio-envoy\") pod \"39c47890-6d91-4569-8e7d-74114231e81c\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " Oct 30 22:14:36.486885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.429642 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39c47890-6d91-4569-8e7d-74114231e81c-kube-api-access-qxgf2\" (UniqueName: \"kubernetes.io/projected/39c47890-6d91-4569-8e7d-74114231e81c-kube-api-access-qxgf2\") pod \"39c47890-6d91-4569-8e7d-74114231e81c\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " Oct 30 22:14:36.486885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.429805 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-credential-socket\") pod \"39c47890-6d91-4569-8e7d-74114231e81c\" (UID: \"39c47890-6d91-4569-8e7d-74114231e81c\") " Oct 30 22:14:36.486885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.433017 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-credential-socket" pod "39c47890-6d91-4569-8e7d-74114231e81c" (UID: "39c47890-6d91-4569-8e7d-74114231e81c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:36.486885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.444432 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-workload-socket" pod "39c47890-6d91-4569-8e7d-74114231e81c" (UID: "39c47890-6d91-4569-8e7d-74114231e81c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:36.514413 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.514374 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:14:36.559419 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.532188 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:14:36.559419 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.532285 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:14:36.559419 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.533189 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:36.559419 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.543870 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 22:14:36.559419 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.548521 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c47890-6d91-4569-8e7d-74114231e81c-istio-token" pod "39c47890-6d91-4569-8e7d-74114231e81c" (UID: "39c47890-6d91-4569-8e7d-74114231e81c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:36.576518 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.559892 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:36.576518 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.576374 6006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Oct 30 22:14:36.577094 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.577065 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-workload-certs" pod "39c47890-6d91-4569-8e7d-74114231e81c" (UID: "39c47890-6d91-4569-8e7d-74114231e81c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:36.634803 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.634759 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39c47890-6d91-4569-8e7d-74114231e81c-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:36.635553 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.635531 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:36.635729 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.635713 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:36.691409 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.656359 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:36.691409 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.656413 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:36.691409 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.656431 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39c47890-6d91-4569-8e7d-74114231e81c-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:36.691409 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.644051 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c47890-6d91-4569-8e7d-74114231e81c-kube-api-access-qxgf2" pod "39c47890-6d91-4569-8e7d-74114231e81c" (UID: "39c47890-6d91-4569-8e7d-74114231e81c"). InnerVolumeSpecName "kube-api-access-qxgf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:36.691409 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.654595 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-istio-envoy" pod "39c47890-6d91-4569-8e7d-74114231e81c" (UID: "39c47890-6d91-4569-8e7d-74114231e81c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:36.691409 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.656675 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:14:36.696404 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.696374 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:14:36.769428 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.769397 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39c47890-6d91-4569-8e7d-74114231e81c-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:36.769637 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.769617 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxgf2\" (UniqueName: \"kubernetes.io/projected/39c47890-6d91-4569-8e7d-74114231e81c-kube-api-access-qxgf2\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:36.805274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.804385 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39c47890-6d91-4569-8e7d-74114231e81c-istio-podinfo" pod "39c47890-6d91-4569-8e7d-74114231e81c" (UID: "39c47890-6d91-4569-8e7d-74114231e81c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:14:36.824595 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.824563 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:14:36.904907 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.878973 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39c47890-6d91-4569-8e7d-74114231e81c-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:36.904907 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.897772 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:36.930275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.922234 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 22:14:36.930275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.922504 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 22:14:36.930666 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.930629 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:36.956272 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:36.931051 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:14:37.137272 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:37.131433 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" Oct 30 22:14:37.137272 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:37.135527 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q" event={"ID":"39c47890-6d91-4569-8e7d-74114231e81c","Type":"ContainerDied","Data":"7424f8687c5506f8ef0c8690acea0d47cbaf5d62b04d01268d746635199a14d9"} Oct 30 22:14:37.137272 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:37.135591 6006 scope.go:122] "RemoveContainer" containerID="c9a749d2e48b6e262628a8a9de7a14510583638c399e3f1571fb566d6a521776" Oct 30 22:14:37.258464 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:37.231114 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:37.505585 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:37.504702 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:14:37.556287 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:37.546450 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:37.625283 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:37.623451 6006 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:14:37.625283 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:37.623544 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" podUID="88f78b1a-a774-44f1-8a64-8786dcdcf528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:37.663724 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:37.663581 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:37.663919 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:37.663820 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 22:14:37.664072 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:37.664056 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:14:37.976658 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:37.976610 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:37.987544 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:37.982651 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:37.987544 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:37.983700 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:38.054893 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:38.038576 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:38.275667 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:38.267548 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:38.439485 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:38.439447 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:14:38.439791 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:38.439773 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:14:38.439995 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:38.439980 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:14:38.440172 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:38.440158 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:14:38.444742 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:38.444719 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:14:38.532240 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:38.531699 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:14:38.569315 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:38.551198 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:14:38.988795 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:38.988745 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q"] Oct 30 22:14:38.994612 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:38.994576 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s5x8q"] Oct 30 22:14:39.607580 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:39.607542 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39c47890-6d91-4569-8e7d-74114231e81c" path="/var/lib/kubelet/pods/39c47890-6d91-4569-8e7d-74114231e81c/volumes" Oct 30 22:14:41.487659 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:41.487620 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9db548fb-k89sh"] Oct 30 22:14:41.488547 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:41.488514 6006 cpu_manager.go:414] "containerMap: removing container" podUID="8f5e21fa-7968-45c9-a8d8-dc086e155f65" containerName="revoke-keys" Oct 30 22:14:41.488547 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:41.488539 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f5e21fa-7968-45c9-a8d8-dc086e155f65" containerName="revoke-keys" Oct 30 22:14:41.488705 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:41.488560 6006 cpu_manager.go:414] "containerMap: removing container" podUID="39c47890-6d91-4569-8e7d-74114231e81c" containerName="istio-proxy" Oct 30 22:14:41.488705 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:41.488569 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39c47890-6d91-4569-8e7d-74114231e81c" containerName="istio-proxy" Oct 30 22:14:41.488705 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:41.488682 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f5e21fa-7968-45c9-a8d8-dc086e155f65" containerName="revoke-keys" Oct 30 22:14:41.488705 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:41.488691 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="39c47890-6d91-4569-8e7d-74114231e81c" containerName="istio-proxy" Oct 30 22:14:41.493838 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:41.493817 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db548fb-k89sh" Oct 30 22:14:41.553188 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:41.553148 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9db548fb-k89sh"] Oct 30 22:14:41.590816 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:41.590779 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:14:41.591026 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:41.590839 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:14:41.596920 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:41.596885 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/605fdfa0-1deb-4036-94c9-7f02f947245b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9db548fb-k89sh\" (UID: \"605fdfa0-1deb-4036-94c9-7f02f947245b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db548fb-k89sh" Oct 30 22:14:41.597098 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:41.596987 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/605fdfa0-1deb-4036-94c9-7f02f947245b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9db548fb-k89sh\" (UID: \"605fdfa0-1deb-4036-94c9-7f02f947245b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db548fb-k89sh" Oct 30 22:14:41.597098 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:41.597063 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt4qp\" (UniqueName: \"kubernetes.io/projected/605fdfa0-1deb-4036-94c9-7f02f947245b-kube-api-access-jt4qp\") pod \"maas-api-revoke-sub-9db548fb-k89sh\" (UID: \"605fdfa0-1deb-4036-94c9-7f02f947245b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db548fb-k89sh" Oct 30 22:14:41.698576 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:41.698534 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/605fdfa0-1deb-4036-94c9-7f02f947245b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9db548fb-k89sh\" (UID: \"605fdfa0-1deb-4036-94c9-7f02f947245b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db548fb-k89sh" Oct 30 22:14:41.698785 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:41.698676 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/605fdfa0-1deb-4036-94c9-7f02f947245b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9db548fb-k89sh\" (UID: \"605fdfa0-1deb-4036-94c9-7f02f947245b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db548fb-k89sh" Oct 30 22:14:41.698785 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:41.698740 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt4qp\" (UniqueName: \"kubernetes.io/projected/605fdfa0-1deb-4036-94c9-7f02f947245b-kube-api-access-jt4qp\") pod \"maas-api-revoke-sub-9db548fb-k89sh\" (UID: \"605fdfa0-1deb-4036-94c9-7f02f947245b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db548fb-k89sh" Oct 30 22:14:41.699635 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:41.699606 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/605fdfa0-1deb-4036-94c9-7f02f947245b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9db548fb-k89sh\" (UID: \"605fdfa0-1deb-4036-94c9-7f02f947245b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db548fb-k89sh" Oct 30 22:14:41.701187 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:41.701159 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/605fdfa0-1deb-4036-94c9-7f02f947245b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9db548fb-k89sh\" (UID: \"605fdfa0-1deb-4036-94c9-7f02f947245b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db548fb-k89sh" Oct 30 22:14:41.757867 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:41.757793 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt4qp\" (UniqueName: \"kubernetes.io/projected/605fdfa0-1deb-4036-94c9-7f02f947245b-kube-api-access-jt4qp\") pod \"maas-api-revoke-sub-9db548fb-k89sh\" (UID: \"605fdfa0-1deb-4036-94c9-7f02f947245b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db548fb-k89sh" Oct 30 22:14:41.810067 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:41.810025 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db548fb-k89sh" Oct 30 22:14:42.194285 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:42.194217 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9db548fb-k89sh"] Oct 30 22:14:42.576266 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:42.576209 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db548fb-k89sh" event={"ID":"605fdfa0-1deb-4036-94c9-7f02f947245b","Type":"ContainerStarted","Data":"9a9d95c02b2fd96eed5098ac39c28fe3d36b42e91b0ea6c826b6deb27e7b893d"} Oct 30 22:14:43.585486 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:43.585445 6006 generic.go:356] "Generic (PLEG): container finished" podID="605fdfa0-1deb-4036-94c9-7f02f947245b" containerID="9fb642f492efab9aec2d2f4584cf984455008c272d641daf398bfa02eb47c1a6" exitCode=0 Oct 30 22:14:43.585899 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:43.585526 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db548fb-k89sh" event={"ID":"605fdfa0-1deb-4036-94c9-7f02f947245b","Type":"ContainerDied","Data":"9fb642f492efab9aec2d2f4584cf984455008c272d641daf398bfa02eb47c1a6"} Oct 30 22:14:44.809009 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:44.808982 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db548fb-k89sh" Oct 30 22:14:44.951705 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:44.951609 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/605fdfa0-1deb-4036-94c9-7f02f947245b-kube-api-access-jt4qp\" (UniqueName: \"kubernetes.io/projected/605fdfa0-1deb-4036-94c9-7f02f947245b-kube-api-access-jt4qp\") pod \"605fdfa0-1deb-4036-94c9-7f02f947245b\" (UID: \"605fdfa0-1deb-4036-94c9-7f02f947245b\") " Oct 30 22:14:44.951705 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:44.951712 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/605fdfa0-1deb-4036-94c9-7f02f947245b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/605fdfa0-1deb-4036-94c9-7f02f947245b-maas-api-openshift-ca\") pod \"605fdfa0-1deb-4036-94c9-7f02f947245b\" (UID: \"605fdfa0-1deb-4036-94c9-7f02f947245b\") " Oct 30 22:14:44.951963 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:44.951876 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/605fdfa0-1deb-4036-94c9-7f02f947245b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/605fdfa0-1deb-4036-94c9-7f02f947245b-maas-api-xks-ca\") pod \"605fdfa0-1deb-4036-94c9-7f02f947245b\" (UID: \"605fdfa0-1deb-4036-94c9-7f02f947245b\") " Oct 30 22:14:44.953341 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:44.953309 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/605fdfa0-1deb-4036-94c9-7f02f947245b-maas-api-openshift-ca" pod "605fdfa0-1deb-4036-94c9-7f02f947245b" (UID: "605fdfa0-1deb-4036-94c9-7f02f947245b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:44.955414 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:44.955385 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/605fdfa0-1deb-4036-94c9-7f02f947245b-maas-api-xks-ca" pod "605fdfa0-1deb-4036-94c9-7f02f947245b" (UID: "605fdfa0-1deb-4036-94c9-7f02f947245b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:44.955610 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:44.955578 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/605fdfa0-1deb-4036-94c9-7f02f947245b-kube-api-access-jt4qp" pod "605fdfa0-1deb-4036-94c9-7f02f947245b" (UID: "605fdfa0-1deb-4036-94c9-7f02f947245b"). InnerVolumeSpecName "kube-api-access-jt4qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:45.053788 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:45.053748 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/605fdfa0-1deb-4036-94c9-7f02f947245b-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:45.053788 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:45.053783 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/605fdfa0-1deb-4036-94c9-7f02f947245b-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:45.053788 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:45.053793 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt4qp\" (UniqueName: \"kubernetes.io/projected/605fdfa0-1deb-4036-94c9-7f02f947245b-kube-api-access-jt4qp\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:45.611183 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:45.611139 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db548fb-k89sh" Oct 30 22:14:45.612964 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:45.612932 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db548fb-k89sh" event={"ID":"605fdfa0-1deb-4036-94c9-7f02f947245b","Type":"ContainerDied","Data":"9a9d95c02b2fd96eed5098ac39c28fe3d36b42e91b0ea6c826b6deb27e7b893d"} Oct 30 22:14:45.613076 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:45.612982 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a9d95c02b2fd96eed5098ac39c28fe3d36b42e91b0ea6c826b6deb27e7b893d" Oct 30 22:14:54.097943 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:54.097901 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c960764-7f6x7"] Oct 30 22:14:54.098619 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:54.098604 6006 cpu_manager.go:414] "containerMap: removing container" podUID="605fdfa0-1deb-4036-94c9-7f02f947245b" containerName="revoke-keys" Oct 30 22:14:54.098670 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:54.098620 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="605fdfa0-1deb-4036-94c9-7f02f947245b" containerName="revoke-keys" Oct 30 22:14:54.098730 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:54.098721 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="605fdfa0-1deb-4036-94c9-7f02f947245b" containerName="revoke-keys" Oct 30 22:14:54.102728 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:54.102709 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c960764-7f6x7" Oct 30 22:14:54.159414 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:54.159366 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c960764-7f6x7"] Oct 30 22:14:54.199722 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:54.199683 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ee862a8-ab50-4538-8f6c-67ff8396afae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c960764-7f6x7\" (UID: \"3ee862a8-ab50-4538-8f6c-67ff8396afae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c960764-7f6x7" Oct 30 22:14:54.199983 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:54.199760 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdwvc\" (UniqueName: \"kubernetes.io/projected/3ee862a8-ab50-4538-8f6c-67ff8396afae-kube-api-access-sdwvc\") pod \"maas-api-revoke-sub-0c960764-7f6x7\" (UID: \"3ee862a8-ab50-4538-8f6c-67ff8396afae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c960764-7f6x7" Oct 30 22:14:54.200064 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:54.200007 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ee862a8-ab50-4538-8f6c-67ff8396afae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c960764-7f6x7\" (UID: \"3ee862a8-ab50-4538-8f6c-67ff8396afae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c960764-7f6x7" Oct 30 22:14:54.200859 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:54.200833 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:14:54.201022 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:54.200998 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:14:54.301122 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:54.301087 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ee862a8-ab50-4538-8f6c-67ff8396afae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c960764-7f6x7\" (UID: \"3ee862a8-ab50-4538-8f6c-67ff8396afae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c960764-7f6x7" Oct 30 22:14:54.301357 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:54.301197 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ee862a8-ab50-4538-8f6c-67ff8396afae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c960764-7f6x7\" (UID: \"3ee862a8-ab50-4538-8f6c-67ff8396afae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c960764-7f6x7" Oct 30 22:14:54.301357 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:54.301267 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdwvc\" (UniqueName: \"kubernetes.io/projected/3ee862a8-ab50-4538-8f6c-67ff8396afae-kube-api-access-sdwvc\") pod \"maas-api-revoke-sub-0c960764-7f6x7\" (UID: \"3ee862a8-ab50-4538-8f6c-67ff8396afae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c960764-7f6x7" Oct 30 22:14:54.302115 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:54.302087 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ee862a8-ab50-4538-8f6c-67ff8396afae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c960764-7f6x7\" (UID: \"3ee862a8-ab50-4538-8f6c-67ff8396afae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c960764-7f6x7" Oct 30 22:14:54.303968 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:54.303920 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ee862a8-ab50-4538-8f6c-67ff8396afae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c960764-7f6x7\" (UID: \"3ee862a8-ab50-4538-8f6c-67ff8396afae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c960764-7f6x7" Oct 30 22:14:54.360931 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:54.360854 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdwvc\" (UniqueName: \"kubernetes.io/projected/3ee862a8-ab50-4538-8f6c-67ff8396afae-kube-api-access-sdwvc\") pod \"maas-api-revoke-sub-0c960764-7f6x7\" (UID: \"3ee862a8-ab50-4538-8f6c-67ff8396afae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c960764-7f6x7" Oct 30 22:14:54.417316 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:54.417282 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c960764-7f6x7" Oct 30 22:14:54.858765 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:54.858729 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c960764-7f6x7"] Oct 30 22:14:54.859313 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:14:54.859283 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee862a8_ab50_4538_8f6c_67ff8396afae.slice/crio-50b1955fdb3b2b73e6481301b37a691bed1b87a072fc0fda622dd88804ca972b WatchSource:0}: Error finding container 50b1955fdb3b2b73e6481301b37a691bed1b87a072fc0fda622dd88804ca972b: Status 404 returned error can't find the container with id 50b1955fdb3b2b73e6481301b37a691bed1b87a072fc0fda622dd88804ca972b Oct 30 22:14:55.691341 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:55.691294 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c960764-7f6x7" event={"ID":"3ee862a8-ab50-4538-8f6c-67ff8396afae","Type":"ContainerStarted","Data":"f318b884d9a900f895d4ddecd1d98ffcf6bf63cecbdaa53c4d1d489872a39d93"} Oct 30 22:14:55.691858 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:55.691368 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c960764-7f6x7" event={"ID":"3ee862a8-ab50-4538-8f6c-67ff8396afae","Type":"ContainerStarted","Data":"50b1955fdb3b2b73e6481301b37a691bed1b87a072fc0fda622dd88804ca972b"} Oct 30 22:14:55.852150 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:55.852070 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c960764-7f6x7" podStartSLOduration=1.8520482999999999 podStartE2EDuration="1.8520483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:55.847716 +0000 UTC m=+1498.399892300" watchObservedRunningTime="2026-10-30 22:14:55.8520483 +0000 UTC m=+1498.404224601" Oct 30 22:14:56.700831 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:56.700795 6006 generic.go:356] "Generic (PLEG): container finished" podID="3ee862a8-ab50-4538-8f6c-67ff8396afae" containerID="f318b884d9a900f895d4ddecd1d98ffcf6bf63cecbdaa53c4d1d489872a39d93" exitCode=0 Oct 30 22:14:56.701326 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:56.700884 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c960764-7f6x7" event={"ID":"3ee862a8-ab50-4538-8f6c-67ff8396afae","Type":"ContainerDied","Data":"f318b884d9a900f895d4ddecd1d98ffcf6bf63cecbdaa53c4d1d489872a39d93"} Oct 30 22:14:57.844464 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:57.844427 6006 scope.go:122] "RemoveContainer" containerID="5ae4cbb8c76b125aff9c3939d66c3100f4eb6644a86fdfc9328363ca1666090b" Oct 30 22:14:57.860073 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:57.860042 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 22:14:57.861123 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:57.861096 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 22:14:57.865007 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:57.864990 6006 scope.go:122] "RemoveContainer" containerID="880ad40d0954770d4886c010099dff90eb6baff768198034546dbc576fd9e1b1" Oct 30 22:14:57.912057 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:57.912031 6006 scope.go:122] "RemoveContainer" containerID="2ddd7e8901c56cc6903f69a9d4833b494c5023eb07b3b70c1fafeece84d5aa8d" Oct 30 22:14:57.924531 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:57.924509 6006 scope.go:122] "RemoveContainer" containerID="f6ae4f6a8e784e01380f575fb36dafef94dc334d941debe33d463155d6473e05" Oct 30 22:14:57.961495 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:14:57.961450 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:14:57.973109 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:57.973082 6006 scope.go:122] "RemoveContainer" containerID="4c8aebbda243f8f92c82dfc893baacfe31f430d2d3d21fed33e36c6d0848e6ca" Oct 30 22:14:58.014678 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:58.014644 6006 scope.go:122] "RemoveContainer" containerID="871bf540b36b8ac79b11b15019f81dad6c29adc3021249f6e077ca5fb516909a" Oct 30 22:14:58.056839 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:58.056813 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c960764-7f6x7" Oct 30 22:14:58.128347 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:58.128314 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ee862a8-ab50-4538-8f6c-67ff8396afae-kube-api-access-sdwvc\" (UniqueName: \"kubernetes.io/projected/3ee862a8-ab50-4538-8f6c-67ff8396afae-kube-api-access-sdwvc\") pod \"3ee862a8-ab50-4538-8f6c-67ff8396afae\" (UID: \"3ee862a8-ab50-4538-8f6c-67ff8396afae\") " Oct 30 22:14:58.128688 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:58.128666 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ee862a8-ab50-4538-8f6c-67ff8396afae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ee862a8-ab50-4538-8f6c-67ff8396afae-maas-api-openshift-ca\") pod \"3ee862a8-ab50-4538-8f6c-67ff8396afae\" (UID: \"3ee862a8-ab50-4538-8f6c-67ff8396afae\") " Oct 30 22:14:58.128782 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:58.128721 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ee862a8-ab50-4538-8f6c-67ff8396afae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ee862a8-ab50-4538-8f6c-67ff8396afae-maas-api-xks-ca\") pod \"3ee862a8-ab50-4538-8f6c-67ff8396afae\" (UID: \"3ee862a8-ab50-4538-8f6c-67ff8396afae\") " Oct 30 22:14:58.129038 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:58.129010 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ee862a8-ab50-4538-8f6c-67ff8396afae-maas-api-openshift-ca" pod "3ee862a8-ab50-4538-8f6c-67ff8396afae" (UID: "3ee862a8-ab50-4538-8f6c-67ff8396afae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:58.129414 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:58.129394 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ee862a8-ab50-4538-8f6c-67ff8396afae-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:58.131013 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:58.130985 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ee862a8-ab50-4538-8f6c-67ff8396afae-kube-api-access-sdwvc" pod "3ee862a8-ab50-4538-8f6c-67ff8396afae" (UID: "3ee862a8-ab50-4538-8f6c-67ff8396afae"). InnerVolumeSpecName "kube-api-access-sdwvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:58.131514 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:58.131488 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ee862a8-ab50-4538-8f6c-67ff8396afae-maas-api-xks-ca" pod "3ee862a8-ab50-4538-8f6c-67ff8396afae" (UID: "3ee862a8-ab50-4538-8f6c-67ff8396afae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:58.231240 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:58.231137 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ee862a8-ab50-4538-8f6c-67ff8396afae-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:58.231240 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:58.231181 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdwvc\" (UniqueName: \"kubernetes.io/projected/3ee862a8-ab50-4538-8f6c-67ff8396afae-kube-api-access-sdwvc\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:58.731163 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:58.731130 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c960764-7f6x7" Oct 30 22:14:58.731392 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:58.731134 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c960764-7f6x7" event={"ID":"3ee862a8-ab50-4538-8f6c-67ff8396afae","Type":"ContainerDied","Data":"50b1955fdb3b2b73e6481301b37a691bed1b87a072fc0fda622dd88804ca972b"} Oct 30 22:14:58.731392 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:14:58.731238 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50b1955fdb3b2b73e6481301b37a691bed1b87a072fc0fda622dd88804ca972b" Oct 30 22:15:00.226771 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.226728 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg"] Oct 30 22:15:00.228457 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.228425 6006 cpu_manager.go:414] "containerMap: removing container" podUID="3ee862a8-ab50-4538-8f6c-67ff8396afae" containerName="revoke-keys" Oct 30 22:15:00.228639 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.228623 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ee862a8-ab50-4538-8f6c-67ff8396afae" containerName="revoke-keys" Oct 30 22:15:00.228912 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.228898 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ee862a8-ab50-4538-8f6c-67ff8396afae" containerName="revoke-keys" Oct 30 22:15:00.238416 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.238380 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm"] Oct 30 22:15:00.238573 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.238477 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg" Oct 30 22:15:00.245462 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.245421 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb"] Oct 30 22:15:00.245664 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.245641 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm" Oct 30 22:15:00.250928 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.250897 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr"] Oct 30 22:15:00.251066 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.251032 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb" Oct 30 22:15:00.256340 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.256315 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr" Oct 30 22:15:00.279357 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.279319 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg"] Oct 30 22:15:00.332865 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.330653 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm"] Oct 30 22:15:00.339901 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.339859 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb"] Oct 30 22:15:00.343264 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.343192 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr"] Oct 30 22:15:00.350949 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.350894 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:15:00.365942 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.365907 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7458\" (UniqueName: \"kubernetes.io/projected/964508d4-8496-4b56-9649-57163fd515ad-kube-api-access-v7458\") pod \"maas-api-key-cleanup-29889975-58bhr\" (UID: \"964508d4-8496-4b56-9649-57163fd515ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr" Oct 30 22:15:00.365942 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.365952 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/964508d4-8496-4b56-9649-57163fd515ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-58bhr\" (UID: \"964508d4-8496-4b56-9649-57163fd515ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr" Oct 30 22:15:00.366231 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.366161 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p2lb\" (UniqueName: \"kubernetes.io/projected/3ad3abe3-f131-4013-adcb-094b495a2cf5-kube-api-access-7p2lb\") pod \"maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg\" (UID: \"3ad3abe3-f131-4013-adcb-094b495a2cf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg" Oct 30 22:15:00.366325 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.366305 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248939eb-01b9-4034-ac58-251c3634087e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm\" (UID: \"248939eb-01b9-4034-ac58-251c3634087e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm" Oct 30 22:15:00.366451 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.366428 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqzzx\" (UniqueName: \"kubernetes.io/projected/248939eb-01b9-4034-ac58-251c3634087e-kube-api-access-dqzzx\") pod \"maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm\" (UID: \"248939eb-01b9-4034-ac58-251c3634087e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm" Oct 30 22:15:00.366550 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.366528 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8wkf\" (UniqueName: \"kubernetes.io/projected/f2f8f7f2-bebe-4525-98aa-e9358bd727d2-kube-api-access-t8wkf\") pod \"maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb\" (UID: \"f2f8f7f2-bebe-4525-98aa-e9358bd727d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb" Oct 30 22:15:00.366634 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.366608 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad3abe3-f131-4013-adcb-094b495a2cf5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg\" (UID: \"3ad3abe3-f131-4013-adcb-094b495a2cf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg" Oct 30 22:15:00.366693 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.366634 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f8f7f2-bebe-4525-98aa-e9358bd727d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb\" (UID: \"f2f8f7f2-bebe-4525-98aa-e9358bd727d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb" Oct 30 22:15:00.468502 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.468466 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/964508d4-8496-4b56-9649-57163fd515ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-58bhr\" (UID: \"964508d4-8496-4b56-9649-57163fd515ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr" Oct 30 22:15:00.468686 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.468583 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p2lb\" (UniqueName: \"kubernetes.io/projected/3ad3abe3-f131-4013-adcb-094b495a2cf5-kube-api-access-7p2lb\") pod \"maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg\" (UID: \"3ad3abe3-f131-4013-adcb-094b495a2cf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg" Oct 30 22:15:00.468686 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.468629 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248939eb-01b9-4034-ac58-251c3634087e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm\" (UID: \"248939eb-01b9-4034-ac58-251c3634087e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm" Oct 30 22:15:00.468769 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.468709 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqzzx\" (UniqueName: \"kubernetes.io/projected/248939eb-01b9-4034-ac58-251c3634087e-kube-api-access-dqzzx\") pod \"maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm\" (UID: \"248939eb-01b9-4034-ac58-251c3634087e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm" Oct 30 22:15:00.468816 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.468767 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8wkf\" (UniqueName: \"kubernetes.io/projected/f2f8f7f2-bebe-4525-98aa-e9358bd727d2-kube-api-access-t8wkf\") pod \"maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb\" (UID: \"f2f8f7f2-bebe-4525-98aa-e9358bd727d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb" Oct 30 22:15:00.468868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.468832 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad3abe3-f131-4013-adcb-094b495a2cf5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg\" (UID: \"3ad3abe3-f131-4013-adcb-094b495a2cf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg" Oct 30 22:15:00.468868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.468856 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f8f7f2-bebe-4525-98aa-e9358bd727d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb\" (UID: \"f2f8f7f2-bebe-4525-98aa-e9358bd727d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb" Oct 30 22:15:00.469034 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.469015 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7458\" (UniqueName: \"kubernetes.io/projected/964508d4-8496-4b56-9649-57163fd515ad-kube-api-access-v7458\") pod \"maas-api-key-cleanup-29889975-58bhr\" (UID: \"964508d4-8496-4b56-9649-57163fd515ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr" Oct 30 22:15:00.469314 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.469286 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/964508d4-8496-4b56-9649-57163fd515ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-58bhr\" (UID: \"964508d4-8496-4b56-9649-57163fd515ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr" Oct 30 22:15:00.469458 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.469440 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248939eb-01b9-4034-ac58-251c3634087e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm\" (UID: \"248939eb-01b9-4034-ac58-251c3634087e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm" Oct 30 22:15:00.469554 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.469537 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f8f7f2-bebe-4525-98aa-e9358bd727d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb\" (UID: \"f2f8f7f2-bebe-4525-98aa-e9358bd727d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb" Oct 30 22:15:00.469592 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.469573 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad3abe3-f131-4013-adcb-094b495a2cf5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg\" (UID: \"3ad3abe3-f131-4013-adcb-094b495a2cf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg" Oct 30 22:15:00.533314 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.533219 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7458\" (UniqueName: \"kubernetes.io/projected/964508d4-8496-4b56-9649-57163fd515ad-kube-api-access-v7458\") pod \"maas-api-key-cleanup-29889975-58bhr\" (UID: \"964508d4-8496-4b56-9649-57163fd515ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr" Oct 30 22:15:00.533487 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.533424 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqzzx\" (UniqueName: \"kubernetes.io/projected/248939eb-01b9-4034-ac58-251c3634087e-kube-api-access-dqzzx\") pod \"maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm\" (UID: \"248939eb-01b9-4034-ac58-251c3634087e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm" Oct 30 22:15:00.533487 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.533424 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8wkf\" (UniqueName: \"kubernetes.io/projected/f2f8f7f2-bebe-4525-98aa-e9358bd727d2-kube-api-access-t8wkf\") pod \"maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb\" (UID: \"f2f8f7f2-bebe-4525-98aa-e9358bd727d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb" Oct 30 22:15:00.536886 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.536850 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p2lb\" (UniqueName: \"kubernetes.io/projected/3ad3abe3-f131-4013-adcb-094b495a2cf5-kube-api-access-7p2lb\") pod \"maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg\" (UID: \"3ad3abe3-f131-4013-adcb-094b495a2cf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg" Oct 30 22:15:00.555297 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.555269 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg" Oct 30 22:15:00.569103 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.569072 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm" Oct 30 22:15:00.584913 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.584882 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb" Oct 30 22:15:00.632804 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.632583 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr" Oct 30 22:15:00.957222 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:00.957186 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg"] Oct 30 22:15:00.957726 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:15:00.957697 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad3abe3_f131_4013_adcb_094b495a2cf5.slice/crio-d310a698f4708730625b1af46138664bb05fb01168c4025daa74f8b9c6964549 WatchSource:0}: Error finding container d310a698f4708730625b1af46138664bb05fb01168c4025daa74f8b9c6964549: Status 404 returned error can't find the container with id d310a698f4708730625b1af46138664bb05fb01168c4025daa74f8b9c6964549 Oct 30 22:15:01.021308 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:01.021107 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm"] Oct 30 22:15:01.021470 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:15:01.021433 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248939eb_01b9_4034_ac58_251c3634087e.slice/crio-5b94875ce83b6b3bc12b64b36dbe9338d777301431bbd8b8cce8168c90023f78 WatchSource:0}: Error finding container 5b94875ce83b6b3bc12b64b36dbe9338d777301431bbd8b8cce8168c90023f78: Status 404 returned error can't find the container with id 5b94875ce83b6b3bc12b64b36dbe9338d777301431bbd8b8cce8168c90023f78 Oct 30 22:15:01.027441 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:01.027036 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb"] Oct 30 22:15:01.029333 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:15:01.029226 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f8f7f2_bebe_4525_98aa_e9358bd727d2.slice/crio-15d33c70b9ca65c7200f936ce53460d6193624d37e2c929c218d1c88c841f1f2 WatchSource:0}: Error finding container 15d33c70b9ca65c7200f936ce53460d6193624d37e2c929c218d1c88c841f1f2: Status 404 returned error can't find the container with id 15d33c70b9ca65c7200f936ce53460d6193624d37e2c929c218d1c88c841f1f2 Oct 30 22:15:01.083808 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:01.083776 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr"] Oct 30 22:15:01.085186 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:15:01.085161 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod964508d4_8496_4b56_9649_57163fd515ad.slice/crio-0224d3dfd04bb22c18616010d40ca451c59131b43811032aa1d66736731ba1d9 WatchSource:0}: Error finding container 0224d3dfd04bb22c18616010d40ca451c59131b43811032aa1d66736731ba1d9: Status 404 returned error can't find the container with id 0224d3dfd04bb22c18616010d40ca451c59131b43811032aa1d66736731ba1d9 Oct 30 22:15:01.765759 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:01.765714 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm" event={"ID":"248939eb-01b9-4034-ac58-251c3634087e","Type":"ContainerStarted","Data":"eaaf499cbe1f3708ea51d278914ae1e625227d899264d2697e5d1d08b8013494"} Oct 30 22:15:01.766241 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:01.765779 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm" event={"ID":"248939eb-01b9-4034-ac58-251c3634087e","Type":"ContainerStarted","Data":"5b94875ce83b6b3bc12b64b36dbe9338d777301431bbd8b8cce8168c90023f78"} Oct 30 22:15:01.767617 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:01.767585 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg" event={"ID":"3ad3abe3-f131-4013-adcb-094b495a2cf5","Type":"ContainerStarted","Data":"e1a02dad42fc1cda0869ed08e604db2e5a06e9fb1a3567601b313c4c92e24ee6"} Oct 30 22:15:01.767767 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:01.767635 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg" event={"ID":"3ad3abe3-f131-4013-adcb-094b495a2cf5","Type":"ContainerStarted","Data":"d310a698f4708730625b1af46138664bb05fb01168c4025daa74f8b9c6964549"} Oct 30 22:15:01.769383 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:01.769357 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb" event={"ID":"f2f8f7f2-bebe-4525-98aa-e9358bd727d2","Type":"ContainerStarted","Data":"fd43907cdfc8515f72368a92dd325788abb40d3b1266b63dff8da82004774489"} Oct 30 22:15:01.769508 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:01.769399 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb" event={"ID":"f2f8f7f2-bebe-4525-98aa-e9358bd727d2","Type":"ContainerStarted","Data":"15d33c70b9ca65c7200f936ce53460d6193624d37e2c929c218d1c88c841f1f2"} Oct 30 22:15:01.771003 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:01.770978 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr" event={"ID":"964508d4-8496-4b56-9649-57163fd515ad","Type":"ContainerStarted","Data":"fcc391add935fc2efd4478efa917cbcbd6368be41abe6fd6ba6bca469444641b"} Oct 30 22:15:01.771125 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:01.771023 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr" event={"ID":"964508d4-8496-4b56-9649-57163fd515ad","Type":"ContainerStarted","Data":"0224d3dfd04bb22c18616010d40ca451c59131b43811032aa1d66736731ba1d9"} Oct 30 22:15:01.914041 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:01.913978 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm" podStartSLOduration=1.913959086 podStartE2EDuration="1.913959086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:01.913444837 +0000 UTC m=+1504.465621152" watchObservedRunningTime="2026-10-30 22:15:01.913959086 +0000 UTC m=+1504.466135387" Oct 30 22:15:02.029239 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.029164 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb" podStartSLOduration=2.029144672 podStartE2EDuration="2.029144672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.024931941 +0000 UTC m=+1504.577108244" watchObservedRunningTime="2026-10-30 22:15:02.029144672 +0000 UTC m=+1504.581320976" Oct 30 22:15:02.143823 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.143767 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg" podStartSLOduration=2.143752542 podStartE2EDuration="2.143752542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.140073735 +0000 UTC m=+1504.692250035" watchObservedRunningTime="2026-10-30 22:15:02.143752542 +0000 UTC m=+1504.695929076" Oct 30 22:15:02.254636 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.254581 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr" podStartSLOduration=2.254566283 podStartE2EDuration="2.254566283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.253094082 +0000 UTC m=+1504.805270382" watchObservedRunningTime="2026-10-30 22:15:02.254566283 +0000 UTC m=+1504.806742588" Oct 30 22:15:02.780664 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.780625 6006 generic.go:356] "Generic (PLEG): container finished" podID="3ad3abe3-f131-4013-adcb-094b495a2cf5" containerID="e1a02dad42fc1cda0869ed08e604db2e5a06e9fb1a3567601b313c4c92e24ee6" exitCode=60 Oct 30 22:15:02.781117 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.780697 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg" event={"ID":"3ad3abe3-f131-4013-adcb-094b495a2cf5","Type":"ContainerDied","Data":"e1a02dad42fc1cda0869ed08e604db2e5a06e9fb1a3567601b313c4c92e24ee6"} Oct 30 22:15:02.781117 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.781059 6006 scope.go:122] "RemoveContainer" containerID="e1a02dad42fc1cda0869ed08e604db2e5a06e9fb1a3567601b313c4c92e24ee6" Oct 30 22:15:02.782703 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.782673 6006 generic.go:356] "Generic (PLEG): container finished" podID="f2f8f7f2-bebe-4525-98aa-e9358bd727d2" containerID="fd43907cdfc8515f72368a92dd325788abb40d3b1266b63dff8da82004774489" exitCode=60 Oct 30 22:15:02.782812 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.782716 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb" event={"ID":"f2f8f7f2-bebe-4525-98aa-e9358bd727d2","Type":"ContainerDied","Data":"fd43907cdfc8515f72368a92dd325788abb40d3b1266b63dff8da82004774489"} Oct 30 22:15:02.783100 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.783049 6006 scope.go:122] "RemoveContainer" containerID="fd43907cdfc8515f72368a92dd325788abb40d3b1266b63dff8da82004774489" Oct 30 22:15:02.784918 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.784631 6006 generic.go:356] "Generic (PLEG): container finished" podID="964508d4-8496-4b56-9649-57163fd515ad" containerID="fcc391add935fc2efd4478efa917cbcbd6368be41abe6fd6ba6bca469444641b" exitCode=60 Oct 30 22:15:02.784918 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.784696 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr" event={"ID":"964508d4-8496-4b56-9649-57163fd515ad","Type":"ContainerDied","Data":"fcc391add935fc2efd4478efa917cbcbd6368be41abe6fd6ba6bca469444641b"} Oct 30 22:15:02.785226 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.784940 6006 scope.go:122] "RemoveContainer" containerID="fcc391add935fc2efd4478efa917cbcbd6368be41abe6fd6ba6bca469444641b" Oct 30 22:15:02.786659 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.786640 6006 generic.go:356] "Generic (PLEG): container finished" podID="248939eb-01b9-4034-ac58-251c3634087e" containerID="eaaf499cbe1f3708ea51d278914ae1e625227d899264d2697e5d1d08b8013494" exitCode=60 Oct 30 22:15:02.786728 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.786679 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm" event={"ID":"248939eb-01b9-4034-ac58-251c3634087e","Type":"ContainerDied","Data":"eaaf499cbe1f3708ea51d278914ae1e625227d899264d2697e5d1d08b8013494"} Oct 30 22:15:02.786927 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.786895 6006 scope.go:122] "RemoveContainer" containerID="eaaf499cbe1f3708ea51d278914ae1e625227d899264d2697e5d1d08b8013494" Oct 30 22:15:02.810547 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.810514 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r"] Oct 30 22:15:02.817918 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.817894 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:02.873798 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.873746 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r"] Oct 30 22:15:02.911524 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.910635 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-envoy\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:02.911524 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.910678 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4rst\" (UniqueName: \"kubernetes.io/projected/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-kube-api-access-x4rst\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:02.911524 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.910705 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istiod-ca-cert\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:02.911524 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.910759 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-data\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:02.911524 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.910814 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-podinfo\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:02.911524 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.910847 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-credential-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:02.911524 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.910872 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-workload-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:02.911524 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.910986 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-workload-certs\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:02.912089 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:02.911836 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-token\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:03.013818 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.013777 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-credential-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:03.014010 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.013835 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-workload-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:03.014010 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.013969 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-workload-certs\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:03.014175 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.014156 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-token\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:03.014875 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.014276 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-credential-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:03.014875 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.014363 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-workload-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:03.014875 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.014650 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-workload-certs\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:03.015112 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.015005 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-envoy\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:03.015112 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.015055 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4rst\" (UniqueName: \"kubernetes.io/projected/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-kube-api-access-x4rst\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:03.015112 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.015106 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istiod-ca-cert\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:03.015566 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.015540 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-data\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:03.015671 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.015609 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-podinfo\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:03.016234 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.016207 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istiod-ca-cert\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:03.016608 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.016582 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-data\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:03.018112 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.018083 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-envoy\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:03.018938 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.018915 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-podinfo\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:03.074840 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.074811 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-token\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:03.075031 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.075010 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4rst\" (UniqueName: \"kubernetes.io/projected/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-kube-api-access-x4rst\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:03.235316 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.216699 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:03.735291 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.735240 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r"] Oct 30 22:15:03.736598 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:15:03.736567 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8bf63b_6ee1_48db_839f_0bd5a6c1743a.slice/crio-5fb8a868e85b3027e014567a5b580bd98268aa51d7f4fec24447b3660d5f443f WatchSource:0}: Error finding container 5fb8a868e85b3027e014567a5b580bd98268aa51d7f4fec24447b3660d5f443f: Status 404 returned error can't find the container with id 5fb8a868e85b3027e014567a5b580bd98268aa51d7f4fec24447b3660d5f443f Oct 30 22:15:03.739491 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.739449 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:15:03.739582 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.739540 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:15:03.739622 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.739596 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:15:03.795487 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.795450 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb" event={"ID":"f2f8f7f2-bebe-4525-98aa-e9358bd727d2","Type":"ContainerStarted","Data":"7299ed2aafdf76d8d0fddc798b60c12aea5d5435941ee39f1fa1aab1180aca94"} Oct 30 22:15:03.797492 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.797461 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr" event={"ID":"964508d4-8496-4b56-9649-57163fd515ad","Type":"ContainerStarted","Data":"91ee14c57ef53314aaf75d938a895a4d0ba9717b6614cc708e65113d4aefb348"} Oct 30 22:15:03.799233 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.799202 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" event={"ID":"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a","Type":"ContainerStarted","Data":"5fb8a868e85b3027e014567a5b580bd98268aa51d7f4fec24447b3660d5f443f"} Oct 30 22:15:03.806799 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.804789 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm" event={"ID":"248939eb-01b9-4034-ac58-251c3634087e","Type":"ContainerStarted","Data":"d57ac9213ad44e615b99bcef232d592d15d7ad65588af4671f75b42b2e86098f"} Oct 30 22:15:03.808408 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:03.808380 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg" event={"ID":"3ad3abe3-f131-4013-adcb-094b495a2cf5","Type":"ContainerStarted","Data":"a2155d278d025dee452f4a0ddb012ed6d201e9848706ba3dcaa31734d2fbf310"} Oct 30 22:15:04.835540 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:04.834562 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" event={"ID":"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a","Type":"ContainerStarted","Data":"ea90b37416b6bd56b2870a5531f4aadd22f4becb74e4a4781146eadae9cc43dc"} Oct 30 22:15:04.857922 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:04.857880 6006 generic.go:356] "Generic (PLEG): container finished" podID="248939eb-01b9-4034-ac58-251c3634087e" containerID="d57ac9213ad44e615b99bcef232d592d15d7ad65588af4671f75b42b2e86098f" exitCode=60 Oct 30 22:15:04.858310 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:04.858285 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm" event={"ID":"248939eb-01b9-4034-ac58-251c3634087e","Type":"ContainerDied","Data":"d57ac9213ad44e615b99bcef232d592d15d7ad65588af4671f75b42b2e86098f"} Oct 30 22:15:04.858486 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:04.858471 6006 scope.go:122] "RemoveContainer" containerID="eaaf499cbe1f3708ea51d278914ae1e625227d899264d2697e5d1d08b8013494" Oct 30 22:15:04.859144 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:04.859125 6006 scope.go:122] "RemoveContainer" containerID="d57ac9213ad44e615b99bcef232d592d15d7ad65588af4671f75b42b2e86098f" Oct 30 22:15:04.862663 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:15:04.862634 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm_odh-ai-gateway-infra(248939eb-01b9-4034-ac58-251c3634087e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm" podUID="248939eb-01b9-4034-ac58-251c3634087e" Oct 30 22:15:04.868314 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:04.868159 6006 generic.go:356] "Generic (PLEG): container finished" podID="3ad3abe3-f131-4013-adcb-094b495a2cf5" containerID="a2155d278d025dee452f4a0ddb012ed6d201e9848706ba3dcaa31734d2fbf310" exitCode=60 Oct 30 22:15:04.869510 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:04.868553 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg" event={"ID":"3ad3abe3-f131-4013-adcb-094b495a2cf5","Type":"ContainerDied","Data":"a2155d278d025dee452f4a0ddb012ed6d201e9848706ba3dcaa31734d2fbf310"} Oct 30 22:15:04.869510 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:04.869071 6006 scope.go:122] "RemoveContainer" containerID="a2155d278d025dee452f4a0ddb012ed6d201e9848706ba3dcaa31734d2fbf310" Oct 30 22:15:04.869510 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:15:04.869437 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg_odh-ai-gateway-infra(3ad3abe3-f131-4013-adcb-094b495a2cf5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg" podUID="3ad3abe3-f131-4013-adcb-094b495a2cf5" Oct 30 22:15:04.874225 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:04.874182 6006 generic.go:356] "Generic (PLEG): container finished" podID="f2f8f7f2-bebe-4525-98aa-e9358bd727d2" containerID="7299ed2aafdf76d8d0fddc798b60c12aea5d5435941ee39f1fa1aab1180aca94" exitCode=60 Oct 30 22:15:04.874697 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:04.874592 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb" event={"ID":"f2f8f7f2-bebe-4525-98aa-e9358bd727d2","Type":"ContainerDied","Data":"7299ed2aafdf76d8d0fddc798b60c12aea5d5435941ee39f1fa1aab1180aca94"} Oct 30 22:15:04.875347 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:04.875317 6006 scope.go:122] "RemoveContainer" containerID="7299ed2aafdf76d8d0fddc798b60c12aea5d5435941ee39f1fa1aab1180aca94" Oct 30 22:15:04.876182 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:15:04.876158 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb_odh-ai-gateway-infra(f2f8f7f2-bebe-4525-98aa-e9358bd727d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb" podUID="f2f8f7f2-bebe-4525-98aa-e9358bd727d2" Oct 30 22:15:04.890183 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:04.890131 6006 generic.go:356] "Generic (PLEG): container finished" podID="964508d4-8496-4b56-9649-57163fd515ad" containerID="91ee14c57ef53314aaf75d938a895a4d0ba9717b6614cc708e65113d4aefb348" exitCode=60 Oct 30 22:15:04.890575 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:04.890545 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr" event={"ID":"964508d4-8496-4b56-9649-57163fd515ad","Type":"ContainerDied","Data":"91ee14c57ef53314aaf75d938a895a4d0ba9717b6614cc708e65113d4aefb348"} Oct 30 22:15:04.891503 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:04.891468 6006 scope.go:122] "RemoveContainer" containerID="91ee14c57ef53314aaf75d938a895a4d0ba9717b6614cc708e65113d4aefb348" Oct 30 22:15:04.892218 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:15:04.892194 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-58bhr_odh-ai-gateway-infra(964508d4-8496-4b56-9649-57163fd515ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr" podUID="964508d4-8496-4b56-9649-57163fd515ad" Oct 30 22:15:04.951099 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:04.951065 6006 scope.go:122] "RemoveContainer" containerID="e1a02dad42fc1cda0869ed08e604db2e5a06e9fb1a3567601b313c4c92e24ee6" Oct 30 22:15:04.988995 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:04.988973 6006 scope.go:122] "RemoveContainer" containerID="fd43907cdfc8515f72368a92dd325788abb40d3b1266b63dff8da82004774489" Oct 30 22:15:05.044898 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:05.044871 6006 scope.go:122] "RemoveContainer" containerID="fcc391add935fc2efd4478efa917cbcbd6368be41abe6fd6ba6bca469444641b" Oct 30 22:15:05.126710 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:05.126574 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" podStartSLOduration=3.126550476 podStartE2EDuration="3.126550476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:04.965552486 +0000 UTC m=+1507.517728787" watchObservedRunningTime="2026-10-30 22:15:05.126550476 +0000 UTC m=+1507.678726777" Oct 30 22:15:05.225276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:05.224649 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:05.230490 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:05.230429 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:15:05.230697 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:05.230514 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" podUID="9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 22:15:05.924441 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:05.924409 6006 scope.go:122] "RemoveContainer" containerID="7299ed2aafdf76d8d0fddc798b60c12aea5d5435941ee39f1fa1aab1180aca94" Oct 30 22:15:05.925510 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:15:05.925476 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb_odh-ai-gateway-infra(f2f8f7f2-bebe-4525-98aa-e9358bd727d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb" podUID="f2f8f7f2-bebe-4525-98aa-e9358bd727d2" Oct 30 22:15:05.934200 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:05.934168 6006 scope.go:122] "RemoveContainer" containerID="91ee14c57ef53314aaf75d938a895a4d0ba9717b6614cc708e65113d4aefb348" Oct 30 22:15:05.943273 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:15:05.934801 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-58bhr_odh-ai-gateway-infra(964508d4-8496-4b56-9649-57163fd515ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr" podUID="964508d4-8496-4b56-9649-57163fd515ad" Oct 30 22:15:05.951044 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:05.951011 6006 scope.go:122] "RemoveContainer" containerID="d57ac9213ad44e615b99bcef232d592d15d7ad65588af4671f75b42b2e86098f" Oct 30 22:15:05.953755 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:15:05.953718 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm_odh-ai-gateway-infra(248939eb-01b9-4034-ac58-251c3634087e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm" podUID="248939eb-01b9-4034-ac58-251c3634087e" Oct 30 22:15:05.954119 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:05.954094 6006 scope.go:122] "RemoveContainer" containerID="a2155d278d025dee452f4a0ddb012ed6d201e9848706ba3dcaa31734d2fbf310" Oct 30 22:15:05.954465 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:15:05.954443 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg_odh-ai-gateway-infra(3ad3abe3-f131-4013-adcb-094b495a2cf5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg" podUID="3ad3abe3-f131-4013-adcb-094b495a2cf5" Oct 30 22:15:06.225461 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:06.222085 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:15:06.225461 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:06.222154 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" podUID="9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 22:15:07.219734 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:07.219691 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:15:07.220297 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:07.219762 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" podUID="9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 22:15:08.221758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:08.221716 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:15:08.222404 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:08.222367 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" podUID="9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 22:15:09.217725 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:09.217679 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:15:09.220341 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:09.218039 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" podUID="9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 22:15:10.277276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:10.234936 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:15:10.277276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:10.235017 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" podUID="9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 22:15:10.901408 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:10.901351 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:11.234855 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:11.234668 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:15:11.235089 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:11.235060 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" podUID="9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 22:15:11.305541 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:11.305493 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:11.307101 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:11.307069 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 22:15:11.308027 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:11.308001 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:11.449403 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:11.449350 6006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:11.518337 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:11.515067 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:11.534384 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:11.531347 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:11.566278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:11.562546 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 22:15:11.713640 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:11.713594 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:11.992296 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:11.992238 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 22:15:12.174279 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:12.169420 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 22:15:12.218134 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:12.194541 6006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": context deadline exceeded" Oct 30 22:15:12.218134 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:12.194705 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:15:12.218134 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:12.196740 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:12.232771 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:12.232464 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:15:12.232771 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:12.232533 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" podUID="9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 22:15:12.266197 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:12.265869 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:15:12.266197 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:12.265959 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" podUID="7cce6f74-3f9a-49cf-a594-211f3bb1f432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:12.316277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:12.306381 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:12.316277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:12.306962 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:15:12.316277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:12.309650 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:12.316277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:12.310079 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 22:15:12.337093 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:12.334576 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:15:12.337093 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:12.334648 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:15:12.459419 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:12.459361 6006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:12.658277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:12.591508 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:12.702061 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:12.702021 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:15:12.702478 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:12.602396 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 22:15:12.720367 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:12.702417 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" podUID="500679e9-0b27-49c3-9b36-7fcfb7d1fc32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:12.720695 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:12.603746 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:12.748505 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:12.703020 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:15:12.748505 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:12.722171 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:15:12.748505 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:12.722221 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:15:12.805305 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:12.780867 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:12.805305 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:12.797406 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:15:12.805305 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:12.802686 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:13.054352 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:13.039414 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 22:15:13.118240 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:13.118207 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:15:13.289474 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:13.289432 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:15:13.289754 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:13.289719 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" podUID="9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 22:15:13.319019 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:13.318915 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 22:15:13.418366 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:13.357087 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 22:15:13.443079 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:13.438364 6006 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:15:13.443079 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:13.438439 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" podUID="46a061d3-8105-4260-9bf1-4fc3a5a09343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:13.443079 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:13.380536 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:13.443079 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:13.360164 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 22:15:13.443079 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:13.438967 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:15:13.443079 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:13.360127 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 22:15:13.606603 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:13.588116 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:15:13.805016 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:13.803515 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 22:15:13.807121 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:13.805273 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:13.807121 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:13.805589 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 22:15:13.809233 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:13.807702 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:13.816635 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:13.816582 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:13.851997 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:13.851806 6006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:14.192817 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:14.166195 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:14.317277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:14.311331 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:15:14.317277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:14.311399 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" podUID="9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 22:15:14.424438 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:14.424372 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 22:15:14.453112 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:14.452995 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:14.469402 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:14.469351 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:15:14.469999 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:14.469965 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:14.595465 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:14.595427 6006 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:15:14.595744 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:14.595698 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" podUID="f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:14.602239 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:14.602205 6006 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:15:14.602511 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:14.602468 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" podUID="ab56f9ed-7e2f-4a74-8dda-53df36c32f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:14.652567 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:15:14.647732 6006 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.03s" Oct 30 22:15:14.652567 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:14.647829 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:15:14.670335 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:14.670303 6006 scope.go:122] "RemoveContainer" containerID="91ee14c57ef53314aaf75d938a895a4d0ba9717b6614cc708e65113d4aefb348" Oct 30 22:15:14.810272 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:14.810209 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:14.810984 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:14.810537 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:14.811923 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:14.810569 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 22:15:14.985553 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:14.810592 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 22:15:14.987601 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:14.827933 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:15:14.992889 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:14.828242 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 22:15:14.996711 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:14.859585 6006 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:15:14.996977 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:14.996939 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" podUID="a852f49d-5ed1-472d-8fb0-f811bbff7e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:15.232275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:15.228775 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:15:15.232600 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:15.232561 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" podUID="9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 22:15:15.425548 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:15.425493 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:15.458716 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:15.458663 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:15.494631 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:15.494579 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:15.525277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:15.495956 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 22:15:15.622339 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:15.618986 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:15.813059 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:15.812426 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 22:15:15.864931 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:15.863651 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 22:15:15.986615 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:15.986565 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:15.994317 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:15.986873 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:16.023665 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:16.003813 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:16.243924 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:16.242111 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 22:15:16.255399 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:16.255360 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:15:16.255668 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:16.255634 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" podUID="9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 22:15:16.494337 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:16.447836 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:16.507351 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:16.471498 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 22:15:16.543209 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:16.536890 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 22:15:16.544737 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:16.540825 6006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Oct 30 22:15:16.562599 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:16.558442 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:16.634303 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:15:16.615000 6006 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.017s" Oct 30 22:15:16.635509 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:16.635482 6006 scope.go:122] "RemoveContainer" containerID="a2155d278d025dee452f4a0ddb012ed6d201e9848706ba3dcaa31734d2fbf310" Oct 30 22:15:16.692347 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:16.676083 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:16.692347 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:16.678064 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:15:16.850282 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:16.823309 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 22:15:16.879273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:16.867305 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:17.023331 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:17.021358 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:17.023828 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:17.023564 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:17.150867 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:17.150823 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 22:15:17.328709 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:17.247355 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:17.328709 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:17.255143 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:15:17.328709 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:17.304968 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" podUID="9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 22:15:17.328709 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:17.305065 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:15:17.530689 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:17.495825 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:17.627034 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:17.546373 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 22:15:17.780561 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:17.546427 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:17.792228 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:17.569746 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:17.792228 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:17.693758 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:17.855283 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:17.853549 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:17.869005 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:17.868937 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:18.090283 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:18.085106 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:18.214782 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:18.160587 6006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Oct 30 22:15:18.215057 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:18.163308 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:18.216116 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:18.216082 6006 scope.go:122] "RemoveContainer" containerID="7299ed2aafdf76d8d0fddc798b60c12aea5d5435941ee39f1fa1aab1180aca94" Oct 30 22:15:18.337860 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:18.337808 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:18.340168 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:18.250782 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:18.340555 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:18.276762 6006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:18.341866 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:18.341592 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 22:15:18.365104 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:18.365075 6006 scope.go:122] "RemoveContainer" containerID="d57ac9213ad44e615b99bcef232d592d15d7ad65588af4671f75b42b2e86098f" Oct 30 22:15:18.839270 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:18.838492 6006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:18.872209 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:18.871076 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:18.897268 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:18.894608 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 22:15:18.914284 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:18.913039 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:15:18.939021 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:18.938986 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:18.944656 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:18.944611 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:19.104277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:19.101646 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 22:15:19.104277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:19.101965 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:19.226505 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:19.226438 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:19.229678 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:19.229314 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:19.269643 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:19.230500 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:15:19.269643 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:19.234385 6006 generic.go:356] "Generic (PLEG): container finished" podID="964508d4-8496-4b56-9649-57163fd515ad" containerID="8b1f9f311d12b52d06472f6440e59b8bd3876df1effb504e9b09fd621f9a074f" exitCode=60 Oct 30 22:15:19.269643 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:19.234470 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:15:19.269643 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:19.234752 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr" event={"ID":"964508d4-8496-4b56-9649-57163fd515ad","Type":"ContainerDied","Data":"8b1f9f311d12b52d06472f6440e59b8bd3876df1effb504e9b09fd621f9a074f"} Oct 30 22:15:19.269643 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:19.234802 6006 scope.go:122] "RemoveContainer" containerID="91ee14c57ef53314aaf75d938a895a4d0ba9717b6614cc708e65113d4aefb348" Oct 30 22:15:19.269643 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:19.237756 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:19.269643 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:19.239114 6006 scope.go:122] "RemoveContainer" containerID="8b1f9f311d12b52d06472f6440e59b8bd3876df1effb504e9b09fd621f9a074f" Oct 30 22:15:19.269643 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:15:19.239434 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-58bhr_odh-ai-gateway-infra(964508d4-8496-4b56-9649-57163fd515ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr" podUID="964508d4-8496-4b56-9649-57163fd515ad" Oct 30 22:15:19.269643 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:19.241844 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:15:19.271710 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:19.271330 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:15:19.271710 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:19.271417 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" podUID="9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:19.304200 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:19.283802 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:15:19.304200 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:19.294085 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:15:19.304200 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:19.295198 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:15:19.304200 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:19.297617 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:15:19.304200 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:19.298854 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:15:19.304200 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:19.299837 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:15:19.307760 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:19.305730 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:15:19.358664 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:19.358555 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:20.257905 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:20.257805 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb" event={"ID":"f2f8f7f2-bebe-4525-98aa-e9358bd727d2","Type":"ContainerStarted","Data":"b6446d53974ca36c6395c27112cb9650ed8db231636858704c4fda4364832bb7"} Oct 30 22:15:20.264179 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:20.264145 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm" event={"ID":"248939eb-01b9-4034-ac58-251c3634087e","Type":"ContainerStarted","Data":"9c5f466d41fe1630b137c08731e5eb373d44d1613b9ff68a53bcd978bd41ac87"} Oct 30 22:15:20.267115 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:20.267083 6006 generic.go:356] "Generic (PLEG): container finished" podID="3ad3abe3-f131-4013-adcb-094b495a2cf5" containerID="23bc3a0ef9c9fc6612e0e5d66500c99b7c4b95a4abad265d8cfcdf3e2586b680" exitCode=60 Oct 30 22:15:20.267321 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:20.267161 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg" event={"ID":"3ad3abe3-f131-4013-adcb-094b495a2cf5","Type":"ContainerDied","Data":"23bc3a0ef9c9fc6612e0e5d66500c99b7c4b95a4abad265d8cfcdf3e2586b680"} Oct 30 22:15:20.267321 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:20.267222 6006 scope.go:122] "RemoveContainer" containerID="a2155d278d025dee452f4a0ddb012ed6d201e9848706ba3dcaa31734d2fbf310" Oct 30 22:15:20.268315 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:20.268190 6006 scope.go:122] "RemoveContainer" containerID="23bc3a0ef9c9fc6612e0e5d66500c99b7c4b95a4abad265d8cfcdf3e2586b680" Oct 30 22:15:20.268792 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:15:20.268603 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg_odh-ai-gateway-infra(3ad3abe3-f131-4013-adcb-094b495a2cf5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg" podUID="3ad3abe3-f131-4013-adcb-094b495a2cf5" Oct 30 22:15:20.789760 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:20.785238 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr"] Oct 30 22:15:20.846195 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:20.846158 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:15:21.072155 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:21.072125 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr" Oct 30 22:15:21.267358 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:21.267314 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/964508d4-8496-4b56-9649-57163fd515ad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/964508d4-8496-4b56-9649-57163fd515ad-maas-api-service-ca\") pod \"964508d4-8496-4b56-9649-57163fd515ad\" (UID: \"964508d4-8496-4b56-9649-57163fd515ad\") " Oct 30 22:15:21.267877 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:21.267652 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/964508d4-8496-4b56-9649-57163fd515ad-kube-api-access-v7458\" (UniqueName: \"kubernetes.io/projected/964508d4-8496-4b56-9649-57163fd515ad-kube-api-access-v7458\") pod \"964508d4-8496-4b56-9649-57163fd515ad\" (UID: \"964508d4-8496-4b56-9649-57163fd515ad\") " Oct 30 22:15:21.274351 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:21.269378 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/964508d4-8496-4b56-9649-57163fd515ad-maas-api-service-ca" pod "964508d4-8496-4b56-9649-57163fd515ad" (UID: "964508d4-8496-4b56-9649-57163fd515ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:21.283990 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:21.283954 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964508d4-8496-4b56-9649-57163fd515ad-kube-api-access-v7458" pod "964508d4-8496-4b56-9649-57163fd515ad" (UID: "964508d4-8496-4b56-9649-57163fd515ad"). InnerVolumeSpecName "kube-api-access-v7458". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:21.364119 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:21.361544 6006 generic.go:356] "Generic (PLEG): container finished" podID="f2f8f7f2-bebe-4525-98aa-e9358bd727d2" containerID="b6446d53974ca36c6395c27112cb9650ed8db231636858704c4fda4364832bb7" exitCode=60 Oct 30 22:15:21.364119 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:21.361607 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb" event={"ID":"f2f8f7f2-bebe-4525-98aa-e9358bd727d2","Type":"ContainerDied","Data":"b6446d53974ca36c6395c27112cb9650ed8db231636858704c4fda4364832bb7"} Oct 30 22:15:21.364119 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:21.362040 6006 scope.go:122] "RemoveContainer" containerID="7299ed2aafdf76d8d0fddc798b60c12aea5d5435941ee39f1fa1aab1180aca94" Oct 30 22:15:21.364119 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:21.362108 6006 scope.go:122] "RemoveContainer" containerID="b6446d53974ca36c6395c27112cb9650ed8db231636858704c4fda4364832bb7" Oct 30 22:15:21.364119 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:15:21.362562 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb_odh-ai-gateway-infra(f2f8f7f2-bebe-4525-98aa-e9358bd727d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb" podUID="f2f8f7f2-bebe-4525-98aa-e9358bd727d2" Oct 30 22:15:21.364119 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:21.364019 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr" event={"ID":"964508d4-8496-4b56-9649-57163fd515ad","Type":"ContainerDied","Data":"0224d3dfd04bb22c18616010d40ca451c59131b43811032aa1d66736731ba1d9"} Oct 30 22:15:21.365794 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:21.364138 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr" Oct 30 22:15:21.370384 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:21.370357 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7458\" (UniqueName: \"kubernetes.io/projected/964508d4-8496-4b56-9649-57163fd515ad-kube-api-access-v7458\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:21.370559 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:21.370544 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/964508d4-8496-4b56-9649-57163fd515ad-maas-api-service-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:21.374448 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:21.373683 6006 generic.go:356] "Generic (PLEG): container finished" podID="248939eb-01b9-4034-ac58-251c3634087e" containerID="9c5f466d41fe1630b137c08731e5eb373d44d1613b9ff68a53bcd978bd41ac87" exitCode=60 Oct 30 22:15:21.374448 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:21.374064 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm" event={"ID":"248939eb-01b9-4034-ac58-251c3634087e","Type":"ContainerDied","Data":"9c5f466d41fe1630b137c08731e5eb373d44d1613b9ff68a53bcd978bd41ac87"} Oct 30 22:15:21.374728 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:21.374648 6006 scope.go:122] "RemoveContainer" containerID="9c5f466d41fe1630b137c08731e5eb373d44d1613b9ff68a53bcd978bd41ac87" Oct 30 22:15:21.375133 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:15:21.375102 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm_odh-ai-gateway-infra(248939eb-01b9-4034-ac58-251c3634087e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm" podUID="248939eb-01b9-4034-ac58-251c3634087e" Oct 30 22:15:21.389747 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:21.389595 6006 scope.go:122] "RemoveContainer" containerID="8b1f9f311d12b52d06472f6440e59b8bd3876df1effb504e9b09fd621f9a074f" Oct 30 22:15:21.401524 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:21.401395 6006 scope.go:122] "RemoveContainer" containerID="d57ac9213ad44e615b99bcef232d592d15d7ad65588af4671f75b42b2e86098f" Oct 30 22:15:21.568369 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:21.566655 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr"] Oct 30 22:15:21.573715 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:21.573675 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-58bhr"] Oct 30 22:15:21.605954 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:21.605912 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="964508d4-8496-4b56-9649-57163fd515ad" path="/var/lib/kubelet/pods/964508d4-8496-4b56-9649-57163fd515ad/volumes" Oct 30 22:15:21.848439 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:21.848401 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm"] Oct 30 22:15:21.972794 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:21.972751 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb"] Oct 30 22:15:22.078984 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.078945 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg"] Oct 30 22:15:22.357092 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.357058 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg" Oct 30 22:15:22.384763 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.384736 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg" Oct 30 22:15:22.384902 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.384767 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg" event={"ID":"3ad3abe3-f131-4013-adcb-094b495a2cf5","Type":"ContainerDied","Data":"d310a698f4708730625b1af46138664bb05fb01168c4025daa74f8b9c6964549"} Oct 30 22:15:22.384902 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.384818 6006 scope.go:122] "RemoveContainer" containerID="23bc3a0ef9c9fc6612e0e5d66500c99b7c4b95a4abad265d8cfcdf3e2586b680" Oct 30 22:15:22.504051 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.504010 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ad3abe3-f131-4013-adcb-094b495a2cf5-kube-api-access-7p2lb\" (UniqueName: \"kubernetes.io/projected/3ad3abe3-f131-4013-adcb-094b495a2cf5-kube-api-access-7p2lb\") pod \"3ad3abe3-f131-4013-adcb-094b495a2cf5\" (UID: \"3ad3abe3-f131-4013-adcb-094b495a2cf5\") " Oct 30 22:15:22.504297 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.504215 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ad3abe3-f131-4013-adcb-094b495a2cf5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad3abe3-f131-4013-adcb-094b495a2cf5-maas-api-service-ca\") pod \"3ad3abe3-f131-4013-adcb-094b495a2cf5\" (UID: \"3ad3abe3-f131-4013-adcb-094b495a2cf5\") " Oct 30 22:15:22.504897 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.504859 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ad3abe3-f131-4013-adcb-094b495a2cf5-maas-api-service-ca" pod "3ad3abe3-f131-4013-adcb-094b495a2cf5" (UID: "3ad3abe3-f131-4013-adcb-094b495a2cf5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:22.505043 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.505018 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad3abe3-f131-4013-adcb-094b495a2cf5-maas-api-service-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:22.507006 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.506973 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad3abe3-f131-4013-adcb-094b495a2cf5-kube-api-access-7p2lb" pod "3ad3abe3-f131-4013-adcb-094b495a2cf5" (UID: "3ad3abe3-f131-4013-adcb-094b495a2cf5"). InnerVolumeSpecName "kube-api-access-7p2lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:22.606807 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.606763 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p2lb\" (UniqueName: \"kubernetes.io/projected/3ad3abe3-f131-4013-adcb-094b495a2cf5-kube-api-access-7p2lb\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:22.664628 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.664596 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb" Oct 30 22:15:22.667513 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.667483 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm" Oct 30 22:15:22.811306 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.811272 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/248939eb-01b9-4034-ac58-251c3634087e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248939eb-01b9-4034-ac58-251c3634087e-maas-api-service-ca\") pod \"248939eb-01b9-4034-ac58-251c3634087e\" (UID: \"248939eb-01b9-4034-ac58-251c3634087e\") " Oct 30 22:15:22.811498 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.811317 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2f8f7f2-bebe-4525-98aa-e9358bd727d2-kube-api-access-t8wkf\" (UniqueName: \"kubernetes.io/projected/f2f8f7f2-bebe-4525-98aa-e9358bd727d2-kube-api-access-t8wkf\") pod \"f2f8f7f2-bebe-4525-98aa-e9358bd727d2\" (UID: \"f2f8f7f2-bebe-4525-98aa-e9358bd727d2\") " Oct 30 22:15:22.811498 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.811397 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/248939eb-01b9-4034-ac58-251c3634087e-kube-api-access-dqzzx\" (UniqueName: \"kubernetes.io/projected/248939eb-01b9-4034-ac58-251c3634087e-kube-api-access-dqzzx\") pod \"248939eb-01b9-4034-ac58-251c3634087e\" (UID: \"248939eb-01b9-4034-ac58-251c3634087e\") " Oct 30 22:15:22.811625 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.811539 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2f8f7f2-bebe-4525-98aa-e9358bd727d2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f8f7f2-bebe-4525-98aa-e9358bd727d2-maas-api-service-ca\") pod \"f2f8f7f2-bebe-4525-98aa-e9358bd727d2\" (UID: \"f2f8f7f2-bebe-4525-98aa-e9358bd727d2\") " Oct 30 22:15:22.811755 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.811727 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/248939eb-01b9-4034-ac58-251c3634087e-maas-api-service-ca" pod "248939eb-01b9-4034-ac58-251c3634087e" (UID: "248939eb-01b9-4034-ac58-251c3634087e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:22.812350 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.812324 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248939eb-01b9-4034-ac58-251c3634087e-maas-api-service-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:22.812520 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.812497 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2f8f7f2-bebe-4525-98aa-e9358bd727d2-maas-api-service-ca" pod "f2f8f7f2-bebe-4525-98aa-e9358bd727d2" (UID: "f2f8f7f2-bebe-4525-98aa-e9358bd727d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:22.814683 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.814659 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/248939eb-01b9-4034-ac58-251c3634087e-kube-api-access-dqzzx" pod "248939eb-01b9-4034-ac58-251c3634087e" (UID: "248939eb-01b9-4034-ac58-251c3634087e"). InnerVolumeSpecName "kube-api-access-dqzzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:22.814903 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.814879 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2f8f7f2-bebe-4525-98aa-e9358bd727d2-kube-api-access-t8wkf" pod "f2f8f7f2-bebe-4525-98aa-e9358bd727d2" (UID: "f2f8f7f2-bebe-4525-98aa-e9358bd727d2"). InnerVolumeSpecName "kube-api-access-t8wkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:22.854118 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.854081 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg"] Oct 30 22:15:22.857908 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.857874 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889975-w22mg"] Oct 30 22:15:22.914513 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.914477 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8wkf\" (UniqueName: \"kubernetes.io/projected/f2f8f7f2-bebe-4525-98aa-e9358bd727d2-kube-api-access-t8wkf\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:22.914513 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.914514 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqzzx\" (UniqueName: \"kubernetes.io/projected/248939eb-01b9-4034-ac58-251c3634087e-kube-api-access-dqzzx\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:22.914723 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:22.914531 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f8f7f2-bebe-4525-98aa-e9358bd727d2-maas-api-service-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:23.398030 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:23.398003 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm" Oct 30 22:15:23.398606 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:23.398035 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm" event={"ID":"248939eb-01b9-4034-ac58-251c3634087e","Type":"ContainerDied","Data":"5b94875ce83b6b3bc12b64b36dbe9338d777301431bbd8b8cce8168c90023f78"} Oct 30 22:15:23.398606 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:23.398108 6006 scope.go:122] "RemoveContainer" containerID="9c5f466d41fe1630b137c08731e5eb373d44d1613b9ff68a53bcd978bd41ac87" Oct 30 22:15:23.401522 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:23.401492 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb" event={"ID":"f2f8f7f2-bebe-4525-98aa-e9358bd727d2","Type":"ContainerDied","Data":"15d33c70b9ca65c7200f936ce53460d6193624d37e2c929c218d1c88c841f1f2"} Oct 30 22:15:23.401682 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:23.401501 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb" Oct 30 22:15:23.412005 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:23.411907 6006 scope.go:122] "RemoveContainer" containerID="b6446d53974ca36c6395c27112cb9650ed8db231636858704c4fda4364832bb7" Oct 30 22:15:23.555623 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:23.555591 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm"] Oct 30 22:15:23.560640 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:23.560612 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bc714d-29889975-tdlpm"] Oct 30 22:15:23.607951 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:23.607860 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="248939eb-01b9-4034-ac58-251c3634087e" path="/var/lib/kubelet/pods/248939eb-01b9-4034-ac58-251c3634087e/volumes" Oct 30 22:15:23.608512 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:23.608486 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ad3abe3-f131-4013-adcb-094b495a2cf5" path="/var/lib/kubelet/pods/3ad3abe3-f131-4013-adcb-094b495a2cf5/volumes" Oct 30 22:15:23.664714 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:23.664679 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb"] Oct 30 22:15:23.670021 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:23.669988 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7bb1c2-29889975-5t7pb"] Oct 30 22:15:25.618346 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:25.618305 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2f8f7f2-bebe-4525-98aa-e9358bd727d2" path="/var/lib/kubelet/pods/f2f8f7f2-bebe-4525-98aa-e9358bd727d2/volumes" Oct 30 22:15:27.911799 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:15:27.911760 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:15:34.019362 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.019315 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn"] Oct 30 22:15:34.020309 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020276 6006 cpu_manager.go:414] "containerMap: removing container" podUID="3ad3abe3-f131-4013-adcb-094b495a2cf5" containerName="cleanup" Oct 30 22:15:34.020309 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020302 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ad3abe3-f131-4013-adcb-094b495a2cf5" containerName="cleanup" Oct 30 22:15:34.020493 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020317 6006 cpu_manager.go:414] "containerMap: removing container" podUID="f2f8f7f2-bebe-4525-98aa-e9358bd727d2" containerName="cleanup" Oct 30 22:15:34.020493 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020322 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2f8f7f2-bebe-4525-98aa-e9358bd727d2" containerName="cleanup" Oct 30 22:15:34.020493 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020333 6006 cpu_manager.go:414] "containerMap: removing container" podUID="964508d4-8496-4b56-9649-57163fd515ad" containerName="cleanup" Oct 30 22:15:34.020493 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020339 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="964508d4-8496-4b56-9649-57163fd515ad" containerName="cleanup" Oct 30 22:15:34.020493 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020349 6006 cpu_manager.go:414] "containerMap: removing container" podUID="3ad3abe3-f131-4013-adcb-094b495a2cf5" containerName="cleanup" Oct 30 22:15:34.020493 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020354 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ad3abe3-f131-4013-adcb-094b495a2cf5" containerName="cleanup" Oct 30 22:15:34.020493 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020383 6006 cpu_manager.go:414] "containerMap: removing container" podUID="964508d4-8496-4b56-9649-57163fd515ad" containerName="cleanup" Oct 30 22:15:34.020493 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020388 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="964508d4-8496-4b56-9649-57163fd515ad" containerName="cleanup" Oct 30 22:15:34.020493 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020395 6006 cpu_manager.go:414] "containerMap: removing container" podUID="964508d4-8496-4b56-9649-57163fd515ad" containerName="cleanup" Oct 30 22:15:34.020493 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020400 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="964508d4-8496-4b56-9649-57163fd515ad" containerName="cleanup" Oct 30 22:15:34.020493 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020409 6006 cpu_manager.go:414] "containerMap: removing container" podUID="248939eb-01b9-4034-ac58-251c3634087e" containerName="cleanup" Oct 30 22:15:34.020493 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020413 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="248939eb-01b9-4034-ac58-251c3634087e" containerName="cleanup" Oct 30 22:15:34.020493 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020425 6006 cpu_manager.go:414] "containerMap: removing container" podUID="248939eb-01b9-4034-ac58-251c3634087e" containerName="cleanup" Oct 30 22:15:34.020493 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020430 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="248939eb-01b9-4034-ac58-251c3634087e" containerName="cleanup" Oct 30 22:15:34.020493 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020446 6006 cpu_manager.go:414] "containerMap: removing container" podUID="f2f8f7f2-bebe-4525-98aa-e9358bd727d2" containerName="cleanup" Oct 30 22:15:34.020493 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020450 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2f8f7f2-bebe-4525-98aa-e9358bd727d2" containerName="cleanup" Oct 30 22:15:34.020493 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020460 6006 cpu_manager.go:414] "containerMap: removing container" podUID="248939eb-01b9-4034-ac58-251c3634087e" containerName="cleanup" Oct 30 22:15:34.020493 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020464 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="248939eb-01b9-4034-ac58-251c3634087e" containerName="cleanup" Oct 30 22:15:34.021310 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020575 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="248939eb-01b9-4034-ac58-251c3634087e" containerName="cleanup" Oct 30 22:15:34.021310 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020587 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="248939eb-01b9-4034-ac58-251c3634087e" containerName="cleanup" Oct 30 22:15:34.021310 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020594 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ad3abe3-f131-4013-adcb-094b495a2cf5" containerName="cleanup" Oct 30 22:15:34.021310 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020601 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2f8f7f2-bebe-4525-98aa-e9358bd727d2" containerName="cleanup" Oct 30 22:15:34.021310 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020608 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ad3abe3-f131-4013-adcb-094b495a2cf5" containerName="cleanup" Oct 30 22:15:34.021310 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020615 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="964508d4-8496-4b56-9649-57163fd515ad" containerName="cleanup" Oct 30 22:15:34.021310 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020620 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ad3abe3-f131-4013-adcb-094b495a2cf5" containerName="cleanup" Oct 30 22:15:34.021310 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020627 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2f8f7f2-bebe-4525-98aa-e9358bd727d2" containerName="cleanup" Oct 30 22:15:34.021310 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020634 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="964508d4-8496-4b56-9649-57163fd515ad" containerName="cleanup" Oct 30 22:15:34.021310 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.020639 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="964508d4-8496-4b56-9649-57163fd515ad" containerName="cleanup" Oct 30 22:15:34.026767 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.026731 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" Oct 30 22:15:34.076346 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.076299 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 22:15:34.076536 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.076361 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 22:15:34.076536 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.076422 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 22:15:34.086875 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.086839 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64dd55d9-83da-4513-b49a-fbca9fc1cd9b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv5vn\" (UID: \"64dd55d9-83da-4513-b49a-fbca9fc1cd9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" Oct 30 22:15:34.087039 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.086978 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/64dd55d9-83da-4513-b49a-fbca9fc1cd9b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv5vn\" (UID: \"64dd55d9-83da-4513-b49a-fbca9fc1cd9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" Oct 30 22:15:34.087039 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.086999 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvlxz\" (UniqueName: \"kubernetes.io/projected/64dd55d9-83da-4513-b49a-fbca9fc1cd9b-kube-api-access-vvlxz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv5vn\" (UID: \"64dd55d9-83da-4513-b49a-fbca9fc1cd9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" Oct 30 22:15:34.087197 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.087169 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/64dd55d9-83da-4513-b49a-fbca9fc1cd9b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv5vn\" (UID: \"64dd55d9-83da-4513-b49a-fbca9fc1cd9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" Oct 30 22:15:34.087278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.087226 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/64dd55d9-83da-4513-b49a-fbca9fc1cd9b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv5vn\" (UID: \"64dd55d9-83da-4513-b49a-fbca9fc1cd9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" Oct 30 22:15:34.087394 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.087375 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/64dd55d9-83da-4513-b49a-fbca9fc1cd9b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv5vn\" (UID: \"64dd55d9-83da-4513-b49a-fbca9fc1cd9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" Oct 30 22:15:34.087516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.087502 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/64dd55d9-83da-4513-b49a-fbca9fc1cd9b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv5vn\" (UID: \"64dd55d9-83da-4513-b49a-fbca9fc1cd9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" Oct 30 22:15:34.123383 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.123345 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wzxhd\"" Oct 30 22:15:34.127635 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.127600 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn"] Oct 30 22:15:34.188955 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.188908 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64dd55d9-83da-4513-b49a-fbca9fc1cd9b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv5vn\" (UID: \"64dd55d9-83da-4513-b49a-fbca9fc1cd9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" Oct 30 22:15:34.189152 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.189037 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/64dd55d9-83da-4513-b49a-fbca9fc1cd9b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv5vn\" (UID: \"64dd55d9-83da-4513-b49a-fbca9fc1cd9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" Oct 30 22:15:34.189152 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.189068 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvlxz\" (UniqueName: \"kubernetes.io/projected/64dd55d9-83da-4513-b49a-fbca9fc1cd9b-kube-api-access-vvlxz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv5vn\" (UID: \"64dd55d9-83da-4513-b49a-fbca9fc1cd9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" Oct 30 22:15:34.189281 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.189166 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/64dd55d9-83da-4513-b49a-fbca9fc1cd9b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv5vn\" (UID: \"64dd55d9-83da-4513-b49a-fbca9fc1cd9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" Oct 30 22:15:34.189281 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.189204 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/64dd55d9-83da-4513-b49a-fbca9fc1cd9b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv5vn\" (UID: \"64dd55d9-83da-4513-b49a-fbca9fc1cd9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" Oct 30 22:15:34.189404 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.189326 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/64dd55d9-83da-4513-b49a-fbca9fc1cd9b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv5vn\" (UID: \"64dd55d9-83da-4513-b49a-fbca9fc1cd9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" Oct 30 22:15:34.189404 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.189400 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/64dd55d9-83da-4513-b49a-fbca9fc1cd9b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv5vn\" (UID: \"64dd55d9-83da-4513-b49a-fbca9fc1cd9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" Oct 30 22:15:34.190029 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.189996 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/64dd55d9-83da-4513-b49a-fbca9fc1cd9b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv5vn\" (UID: \"64dd55d9-83da-4513-b49a-fbca9fc1cd9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" Oct 30 22:15:34.191800 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.191762 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/64dd55d9-83da-4513-b49a-fbca9fc1cd9b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv5vn\" (UID: \"64dd55d9-83da-4513-b49a-fbca9fc1cd9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" Oct 30 22:15:34.191927 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.191844 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/64dd55d9-83da-4513-b49a-fbca9fc1cd9b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv5vn\" (UID: \"64dd55d9-83da-4513-b49a-fbca9fc1cd9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" Oct 30 22:15:34.191997 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.191925 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/64dd55d9-83da-4513-b49a-fbca9fc1cd9b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv5vn\" (UID: \"64dd55d9-83da-4513-b49a-fbca9fc1cd9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" Oct 30 22:15:34.191997 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.191984 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/64dd55d9-83da-4513-b49a-fbca9fc1cd9b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv5vn\" (UID: \"64dd55d9-83da-4513-b49a-fbca9fc1cd9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" Oct 30 22:15:34.247645 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.247608 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64dd55d9-83da-4513-b49a-fbca9fc1cd9b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv5vn\" (UID: \"64dd55d9-83da-4513-b49a-fbca9fc1cd9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" Oct 30 22:15:34.247906 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.247882 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvlxz\" (UniqueName: \"kubernetes.io/projected/64dd55d9-83da-4513-b49a-fbca9fc1cd9b-kube-api-access-vvlxz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kv5vn\" (UID: \"64dd55d9-83da-4513-b49a-fbca9fc1cd9b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" Oct 30 22:15:34.342455 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.342416 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" Oct 30 22:15:34.774412 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.774381 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn"] Oct 30 22:15:34.774715 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:15:34.774691 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64dd55d9_83da_4513_b49a_fbca9fc1cd9b.slice/crio-2e21ffbf321c8210fb68082bb1c5ab20b4d0356c9cf435feac29e10e9c4a5a2a WatchSource:0}: Error finding container 2e21ffbf321c8210fb68082bb1c5ab20b4d0356c9cf435feac29e10e9c4a5a2a: Status 404 returned error can't find the container with id 2e21ffbf321c8210fb68082bb1c5ab20b4d0356c9cf435feac29e10e9c4a5a2a Oct 30 22:15:34.776875 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:34.776850 6006 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:15:35.504549 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:35.504507 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" event={"ID":"64dd55d9-83da-4513-b49a-fbca9fc1cd9b","Type":"ContainerStarted","Data":"2e21ffbf321c8210fb68082bb1c5ab20b4d0356c9cf435feac29e10e9c4a5a2a"} Oct 30 22:15:38.721529 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:38.721495 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:15:38.721820 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:38.721565 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:15:39.546048 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:39.545998 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" event={"ID":"64dd55d9-83da-4513-b49a-fbca9fc1cd9b","Type":"ContainerStarted","Data":"7e2a2384d24bd0cd34ee5b99a3016f1513ae12037ed9f145e47a4a593e5038a3"} Oct 30 22:15:39.546271 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:39.546240 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" Oct 30 22:15:39.548778 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:39.548735 6006 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kv5vn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:15:39.548911 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:39.548804 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" podUID="64dd55d9-83da-4513-b49a-fbca9fc1cd9b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:15:39.760479 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:39.760410 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" podStartSLOduration=2.817363137 podStartE2EDuration="6.76039427s" totalImagesPullingTime="3.943031133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:39.710657846 +0000 UTC m=+1542.262834158" watchObservedRunningTime="2026-10-30 22:15:39.76039427 +0000 UTC m=+1542.312570570" Oct 30 22:15:40.555313 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:40.555273 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kv5vn" Oct 30 22:15:48.607504 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:48.607464 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:15:49.941709 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:49.941613 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-374a479d-gnb7w"] Oct 30 22:15:49.943263 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:49.943222 6006 cpu_manager.go:414] "containerMap: removing container" podUID="f2f8f7f2-bebe-4525-98aa-e9358bd727d2" containerName="cleanup" Oct 30 22:15:49.943423 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:49.943406 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2f8f7f2-bebe-4525-98aa-e9358bd727d2" containerName="cleanup" Oct 30 22:15:49.943518 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:49.943508 6006 cpu_manager.go:414] "containerMap: removing container" podUID="3ad3abe3-f131-4013-adcb-094b495a2cf5" containerName="cleanup" Oct 30 22:15:49.943589 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:49.943579 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ad3abe3-f131-4013-adcb-094b495a2cf5" containerName="cleanup" Oct 30 22:15:49.943913 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:49.943901 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="248939eb-01b9-4034-ac58-251c3634087e" containerName="cleanup" Oct 30 22:15:49.943994 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:49.943985 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2f8f7f2-bebe-4525-98aa-e9358bd727d2" containerName="cleanup" Oct 30 22:15:49.966037 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:49.964798 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374a479d-gnb7w" Oct 30 22:15:49.998171 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:49.998133 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-374a479d-gnb7w"] Oct 30 22:15:50.041669 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:50.041610 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:15:50.067387 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:50.067329 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:15:50.086421 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:50.085735 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3e047c1-c154-4169-b4b4-79849a7f8f1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-374a479d-gnb7w\" (UID: \"c3e047c1-c154-4169-b4b4-79849a7f8f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374a479d-gnb7w" Oct 30 22:15:50.086421 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:50.086011 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgpvr\" (UniqueName: \"kubernetes.io/projected/c3e047c1-c154-4169-b4b4-79849a7f8f1e-kube-api-access-xgpvr\") pod \"maas-api-revoke-sub-374a479d-gnb7w\" (UID: \"c3e047c1-c154-4169-b4b4-79849a7f8f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374a479d-gnb7w" Oct 30 22:15:50.086421 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:50.086209 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3e047c1-c154-4169-b4b4-79849a7f8f1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-374a479d-gnb7w\" (UID: \"c3e047c1-c154-4169-b4b4-79849a7f8f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374a479d-gnb7w" Oct 30 22:15:50.242030 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:50.240627 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3e047c1-c154-4169-b4b4-79849a7f8f1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-374a479d-gnb7w\" (UID: \"c3e047c1-c154-4169-b4b4-79849a7f8f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374a479d-gnb7w" Oct 30 22:15:50.242030 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:50.240787 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3e047c1-c154-4169-b4b4-79849a7f8f1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-374a479d-gnb7w\" (UID: \"c3e047c1-c154-4169-b4b4-79849a7f8f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374a479d-gnb7w" Oct 30 22:15:50.242030 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:50.240965 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgpvr\" (UniqueName: \"kubernetes.io/projected/c3e047c1-c154-4169-b4b4-79849a7f8f1e-kube-api-access-xgpvr\") pod \"maas-api-revoke-sub-374a479d-gnb7w\" (UID: \"c3e047c1-c154-4169-b4b4-79849a7f8f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374a479d-gnb7w" Oct 30 22:15:50.251272 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:50.247097 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3e047c1-c154-4169-b4b4-79849a7f8f1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-374a479d-gnb7w\" (UID: \"c3e047c1-c154-4169-b4b4-79849a7f8f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374a479d-gnb7w" Oct 30 22:15:50.443755 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:50.443275 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgpvr\" (UniqueName: \"kubernetes.io/projected/c3e047c1-c154-4169-b4b4-79849a7f8f1e-kube-api-access-xgpvr\") pod \"maas-api-revoke-sub-374a479d-gnb7w\" (UID: \"c3e047c1-c154-4169-b4b4-79849a7f8f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374a479d-gnb7w" Oct 30 22:15:50.490766 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:50.490690 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3e047c1-c154-4169-b4b4-79849a7f8f1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-374a479d-gnb7w\" (UID: \"c3e047c1-c154-4169-b4b4-79849a7f8f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374a479d-gnb7w" Oct 30 22:15:50.688274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:50.661212 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374a479d-gnb7w" Oct 30 22:15:51.448384 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:51.448317 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:51.448384 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:51.448355 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 22:15:51.828466 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:51.762362 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:51.828466 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:51.806463 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:51.861276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:51.835526 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:52.230066 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:52.029262 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:52.230066 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:15:52.197922 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e047c1_c154_4169_b4b4_79849a7f8f1e.slice/crio-c8a3dfe303ca2661e2abdd67362ee910dfa66e24b6bc14c0ca265539f35cba5f WatchSource:0}: Error finding container c8a3dfe303ca2661e2abdd67362ee910dfa66e24b6bc14c0ca265539f35cba5f: Status 404 returned error can't find the container with id c8a3dfe303ca2661e2abdd67362ee910dfa66e24b6bc14c0ca265539f35cba5f Oct 30 22:15:52.300283 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:52.296937 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-374a479d-gnb7w"] Oct 30 22:15:53.057040 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:53.056989 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374a479d-gnb7w" event={"ID":"c3e047c1-c154-4169-b4b4-79849a7f8f1e","Type":"ContainerStarted","Data":"7d4d39b1a235523ce43c63c6b59e5207dabdf45f9608acf4140c68f93c42e9e2"} Oct 30 22:15:53.057552 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:53.057068 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374a479d-gnb7w" event={"ID":"c3e047c1-c154-4169-b4b4-79849a7f8f1e","Type":"ContainerStarted","Data":"c8a3dfe303ca2661e2abdd67362ee910dfa66e24b6bc14c0ca265539f35cba5f"} Oct 30 22:15:53.169969 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:53.169857 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374a479d-gnb7w" podStartSLOduration=4.169839765 podStartE2EDuration="4.169839765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:53.167863355 +0000 UTC m=+1555.720039657" watchObservedRunningTime="2026-10-30 22:15:53.169839765 +0000 UTC m=+1555.722016122" Oct 30 22:15:54.066602 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:54.066561 6006 generic.go:356] "Generic (PLEG): container finished" podID="c3e047c1-c154-4169-b4b4-79849a7f8f1e" containerID="7d4d39b1a235523ce43c63c6b59e5207dabdf45f9608acf4140c68f93c42e9e2" exitCode=0 Oct 30 22:15:54.067011 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:54.066652 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374a479d-gnb7w" event={"ID":"c3e047c1-c154-4169-b4b4-79849a7f8f1e","Type":"ContainerDied","Data":"7d4d39b1a235523ce43c63c6b59e5207dabdf45f9608acf4140c68f93c42e9e2"} Oct 30 22:15:55.253265 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:55.253217 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374a479d-gnb7w" Oct 30 22:15:55.313596 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:55.313557 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3e047c1-c154-4169-b4b4-79849a7f8f1e-kube-api-access-xgpvr\" (UniqueName: \"kubernetes.io/projected/c3e047c1-c154-4169-b4b4-79849a7f8f1e-kube-api-access-xgpvr\") pod \"c3e047c1-c154-4169-b4b4-79849a7f8f1e\" (UID: \"c3e047c1-c154-4169-b4b4-79849a7f8f1e\") " Oct 30 22:15:55.313819 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:55.313805 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3e047c1-c154-4169-b4b4-79849a7f8f1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3e047c1-c154-4169-b4b4-79849a7f8f1e-maas-api-openshift-ca\") pod \"c3e047c1-c154-4169-b4b4-79849a7f8f1e\" (UID: \"c3e047c1-c154-4169-b4b4-79849a7f8f1e\") " Oct 30 22:15:55.313916 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:55.313901 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3e047c1-c154-4169-b4b4-79849a7f8f1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3e047c1-c154-4169-b4b4-79849a7f8f1e-maas-api-xks-ca\") pod \"c3e047c1-c154-4169-b4b4-79849a7f8f1e\" (UID: \"c3e047c1-c154-4169-b4b4-79849a7f8f1e\") " Oct 30 22:15:55.314304 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:55.314237 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3e047c1-c154-4169-b4b4-79849a7f8f1e-maas-api-openshift-ca" pod "c3e047c1-c154-4169-b4b4-79849a7f8f1e" (UID: "c3e047c1-c154-4169-b4b4-79849a7f8f1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:55.314574 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:55.314553 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3e047c1-c154-4169-b4b4-79849a7f8f1e-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:55.316187 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:55.316153 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3e047c1-c154-4169-b4b4-79849a7f8f1e-maas-api-xks-ca" pod "c3e047c1-c154-4169-b4b4-79849a7f8f1e" (UID: "c3e047c1-c154-4169-b4b4-79849a7f8f1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:55.316323 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:55.316299 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e047c1-c154-4169-b4b4-79849a7f8f1e-kube-api-access-xgpvr" pod "c3e047c1-c154-4169-b4b4-79849a7f8f1e" (UID: "c3e047c1-c154-4169-b4b4-79849a7f8f1e"). InnerVolumeSpecName "kube-api-access-xgpvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:55.416454 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:55.416412 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgpvr\" (UniqueName: \"kubernetes.io/projected/c3e047c1-c154-4169-b4b4-79849a7f8f1e-kube-api-access-xgpvr\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:55.416454 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:55.416449 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3e047c1-c154-4169-b4b4-79849a7f8f1e-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:56.082758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:56.082720 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374a479d-gnb7w" event={"ID":"c3e047c1-c154-4169-b4b4-79849a7f8f1e","Type":"ContainerDied","Data":"c8a3dfe303ca2661e2abdd67362ee910dfa66e24b6bc14c0ca265539f35cba5f"} Oct 30 22:15:56.082758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:56.082758 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374a479d-gnb7w" Oct 30 22:15:56.083010 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:56.082773 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8a3dfe303ca2661e2abdd67362ee910dfa66e24b6bc14c0ca265539f35cba5f" Oct 30 22:15:57.923723 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:15:57.923679 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:15:58.114312 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:15:58.114244 6006 scope.go:122] "RemoveContainer" containerID="26579db3a73415884ae9cae4c6c84a10bdca4aad6bbf6847b0970ba33513c4ae" Oct 30 22:16:09.188816 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:09.188773 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e126c47b-t6vw7"] Oct 30 22:16:09.189819 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:09.189798 6006 cpu_manager.go:414] "containerMap: removing container" podUID="c3e047c1-c154-4169-b4b4-79849a7f8f1e" containerName="revoke-keys" Oct 30 22:16:09.189874 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:09.189823 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3e047c1-c154-4169-b4b4-79849a7f8f1e" containerName="revoke-keys" Oct 30 22:16:09.190035 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:09.190024 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3e047c1-c154-4169-b4b4-79849a7f8f1e" containerName="revoke-keys" Oct 30 22:16:09.199415 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:09.199391 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e126c47b-t6vw7" Oct 30 22:16:09.249278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:09.249212 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e126c47b-t6vw7"] Oct 30 22:16:09.274797 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:09.274757 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a160973-4fcc-4efa-ba15-fe94f2a8cbd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e126c47b-t6vw7\" (UID: \"8a160973-4fcc-4efa-ba15-fe94f2a8cbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e126c47b-t6vw7" Oct 30 22:16:09.275054 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:09.275027 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a160973-4fcc-4efa-ba15-fe94f2a8cbd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e126c47b-t6vw7\" (UID: \"8a160973-4fcc-4efa-ba15-fe94f2a8cbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e126c47b-t6vw7" Oct 30 22:16:09.275333 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:09.275297 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk2h6\" (UniqueName: \"kubernetes.io/projected/8a160973-4fcc-4efa-ba15-fe94f2a8cbd8-kube-api-access-lk2h6\") pod \"maas-api-revoke-sub-e126c47b-t6vw7\" (UID: \"8a160973-4fcc-4efa-ba15-fe94f2a8cbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e126c47b-t6vw7" Oct 30 22:16:09.283156 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:09.283125 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:16:09.290641 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:09.290619 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:16:09.377951 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:09.377905 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a160973-4fcc-4efa-ba15-fe94f2a8cbd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e126c47b-t6vw7\" (UID: \"8a160973-4fcc-4efa-ba15-fe94f2a8cbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e126c47b-t6vw7" Oct 30 22:16:09.378499 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:09.378467 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk2h6\" (UniqueName: \"kubernetes.io/projected/8a160973-4fcc-4efa-ba15-fe94f2a8cbd8-kube-api-access-lk2h6\") pod \"maas-api-revoke-sub-e126c47b-t6vw7\" (UID: \"8a160973-4fcc-4efa-ba15-fe94f2a8cbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e126c47b-t6vw7" Oct 30 22:16:09.378666 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:09.378550 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a160973-4fcc-4efa-ba15-fe94f2a8cbd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e126c47b-t6vw7\" (UID: \"8a160973-4fcc-4efa-ba15-fe94f2a8cbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e126c47b-t6vw7" Oct 30 22:16:09.379262 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:09.379222 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a160973-4fcc-4efa-ba15-fe94f2a8cbd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e126c47b-t6vw7\" (UID: \"8a160973-4fcc-4efa-ba15-fe94f2a8cbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e126c47b-t6vw7" Oct 30 22:16:09.380689 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:09.380662 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a160973-4fcc-4efa-ba15-fe94f2a8cbd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e126c47b-t6vw7\" (UID: \"8a160973-4fcc-4efa-ba15-fe94f2a8cbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e126c47b-t6vw7" Oct 30 22:16:09.437201 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:09.437165 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk2h6\" (UniqueName: \"kubernetes.io/projected/8a160973-4fcc-4efa-ba15-fe94f2a8cbd8-kube-api-access-lk2h6\") pod \"maas-api-revoke-sub-e126c47b-t6vw7\" (UID: \"8a160973-4fcc-4efa-ba15-fe94f2a8cbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e126c47b-t6vw7" Oct 30 22:16:09.514398 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:09.514313 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e126c47b-t6vw7" Oct 30 22:16:09.934866 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:09.934833 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e126c47b-t6vw7"] Oct 30 22:16:09.936350 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:16:09.936323 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a160973_4fcc_4efa_ba15_fe94f2a8cbd8.slice/crio-e48899e5f3d6616e812551474802c7ac51faaf37c23b400a7952602f45a6fe87 WatchSource:0}: Error finding container e48899e5f3d6616e812551474802c7ac51faaf37c23b400a7952602f45a6fe87: Status 404 returned error can't find the container with id e48899e5f3d6616e812551474802c7ac51faaf37c23b400a7952602f45a6fe87 Oct 30 22:16:10.202889 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:10.202854 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e126c47b-t6vw7" event={"ID":"8a160973-4fcc-4efa-ba15-fe94f2a8cbd8","Type":"ContainerStarted","Data":"e48899e5f3d6616e812551474802c7ac51faaf37c23b400a7952602f45a6fe87"} Oct 30 22:16:11.213358 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:11.213261 6006 generic.go:356] "Generic (PLEG): container finished" podID="8a160973-4fcc-4efa-ba15-fe94f2a8cbd8" containerID="7fd53d6debbf81df22723f873cfadda0f91d4eed91e9f9a99e93caa155b37982" exitCode=0 Oct 30 22:16:11.213734 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:11.213366 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e126c47b-t6vw7" event={"ID":"8a160973-4fcc-4efa-ba15-fe94f2a8cbd8","Type":"ContainerDied","Data":"7fd53d6debbf81df22723f873cfadda0f91d4eed91e9f9a99e93caa155b37982"} Oct 30 22:16:12.411206 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:12.411175 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e126c47b-t6vw7" Oct 30 22:16:12.532791 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:12.532694 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a160973-4fcc-4efa-ba15-fe94f2a8cbd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a160973-4fcc-4efa-ba15-fe94f2a8cbd8-maas-api-openshift-ca\") pod \"8a160973-4fcc-4efa-ba15-fe94f2a8cbd8\" (UID: \"8a160973-4fcc-4efa-ba15-fe94f2a8cbd8\") " Oct 30 22:16:12.532966 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:12.532799 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a160973-4fcc-4efa-ba15-fe94f2a8cbd8-kube-api-access-lk2h6\" (UniqueName: \"kubernetes.io/projected/8a160973-4fcc-4efa-ba15-fe94f2a8cbd8-kube-api-access-lk2h6\") pod \"8a160973-4fcc-4efa-ba15-fe94f2a8cbd8\" (UID: \"8a160973-4fcc-4efa-ba15-fe94f2a8cbd8\") " Oct 30 22:16:12.533046 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:12.532967 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a160973-4fcc-4efa-ba15-fe94f2a8cbd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a160973-4fcc-4efa-ba15-fe94f2a8cbd8-maas-api-xks-ca\") pod \"8a160973-4fcc-4efa-ba15-fe94f2a8cbd8\" (UID: \"8a160973-4fcc-4efa-ba15-fe94f2a8cbd8\") " Oct 30 22:16:12.533190 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:12.533161 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a160973-4fcc-4efa-ba15-fe94f2a8cbd8-maas-api-openshift-ca" pod "8a160973-4fcc-4efa-ba15-fe94f2a8cbd8" (UID: "8a160973-4fcc-4efa-ba15-fe94f2a8cbd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:12.533838 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:12.533802 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a160973-4fcc-4efa-ba15-fe94f2a8cbd8-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:12.535621 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:12.535591 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a160973-4fcc-4efa-ba15-fe94f2a8cbd8-kube-api-access-lk2h6" pod "8a160973-4fcc-4efa-ba15-fe94f2a8cbd8" (UID: "8a160973-4fcc-4efa-ba15-fe94f2a8cbd8"). InnerVolumeSpecName "kube-api-access-lk2h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:12.536067 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:12.536042 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a160973-4fcc-4efa-ba15-fe94f2a8cbd8-maas-api-xks-ca" pod "8a160973-4fcc-4efa-ba15-fe94f2a8cbd8" (UID: "8a160973-4fcc-4efa-ba15-fe94f2a8cbd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:12.635537 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:12.635501 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk2h6\" (UniqueName: \"kubernetes.io/projected/8a160973-4fcc-4efa-ba15-fe94f2a8cbd8-kube-api-access-lk2h6\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:12.635537 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:12.635532 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a160973-4fcc-4efa-ba15-fe94f2a8cbd8-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:13.236056 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:13.236011 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e126c47b-t6vw7" event={"ID":"8a160973-4fcc-4efa-ba15-fe94f2a8cbd8","Type":"ContainerDied","Data":"e48899e5f3d6616e812551474802c7ac51faaf37c23b400a7952602f45a6fe87"} Oct 30 22:16:13.236056 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:13.236037 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e126c47b-t6vw7" Oct 30 22:16:13.236300 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:13.236091 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e48899e5f3d6616e812551474802c7ac51faaf37c23b400a7952602f45a6fe87" Oct 30 22:16:20.007882 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:20.007842 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df6ce9a7-7jhmw"] Oct 30 22:16:20.008528 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:20.008513 6006 cpu_manager.go:414] "containerMap: removing container" podUID="8a160973-4fcc-4efa-ba15-fe94f2a8cbd8" containerName="revoke-keys" Oct 30 22:16:20.008592 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:20.008529 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a160973-4fcc-4efa-ba15-fe94f2a8cbd8" containerName="revoke-keys" Oct 30 22:16:20.008684 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:20.008674 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a160973-4fcc-4efa-ba15-fe94f2a8cbd8" containerName="revoke-keys" Oct 30 22:16:20.012413 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:20.012393 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ce9a7-7jhmw" Oct 30 22:16:20.069343 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:20.069302 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df6ce9a7-7jhmw"] Oct 30 22:16:20.111091 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:20.111058 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:16:20.111091 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:20.111097 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:16:20.160803 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:20.160766 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3dd5471-c0c9-42c2-bed5-1edb497c0014-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df6ce9a7-7jhmw\" (UID: \"c3dd5471-c0c9-42c2-bed5-1edb497c0014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ce9a7-7jhmw" Oct 30 22:16:20.161026 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:20.161005 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3dd5471-c0c9-42c2-bed5-1edb497c0014-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df6ce9a7-7jhmw\" (UID: \"c3dd5471-c0c9-42c2-bed5-1edb497c0014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ce9a7-7jhmw" Oct 30 22:16:20.161145 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:20.161130 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4v8v\" (UniqueName: \"kubernetes.io/projected/c3dd5471-c0c9-42c2-bed5-1edb497c0014-kube-api-access-l4v8v\") pod \"maas-api-revoke-sub-df6ce9a7-7jhmw\" (UID: \"c3dd5471-c0c9-42c2-bed5-1edb497c0014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ce9a7-7jhmw" Oct 30 22:16:20.262962 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:20.262864 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4v8v\" (UniqueName: \"kubernetes.io/projected/c3dd5471-c0c9-42c2-bed5-1edb497c0014-kube-api-access-l4v8v\") pod \"maas-api-revoke-sub-df6ce9a7-7jhmw\" (UID: \"c3dd5471-c0c9-42c2-bed5-1edb497c0014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ce9a7-7jhmw" Oct 30 22:16:20.262962 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:20.262954 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3dd5471-c0c9-42c2-bed5-1edb497c0014-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df6ce9a7-7jhmw\" (UID: \"c3dd5471-c0c9-42c2-bed5-1edb497c0014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ce9a7-7jhmw" Oct 30 22:16:20.263223 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:20.263105 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3dd5471-c0c9-42c2-bed5-1edb497c0014-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df6ce9a7-7jhmw\" (UID: \"c3dd5471-c0c9-42c2-bed5-1edb497c0014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ce9a7-7jhmw" Oct 30 22:16:20.263665 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:20.263644 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3dd5471-c0c9-42c2-bed5-1edb497c0014-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df6ce9a7-7jhmw\" (UID: \"c3dd5471-c0c9-42c2-bed5-1edb497c0014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ce9a7-7jhmw" Oct 30 22:16:20.265511 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:20.265490 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3dd5471-c0c9-42c2-bed5-1edb497c0014-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df6ce9a7-7jhmw\" (UID: \"c3dd5471-c0c9-42c2-bed5-1edb497c0014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ce9a7-7jhmw" Oct 30 22:16:20.322233 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:20.322197 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4v8v\" (UniqueName: \"kubernetes.io/projected/c3dd5471-c0c9-42c2-bed5-1edb497c0014-kube-api-access-l4v8v\") pod \"maas-api-revoke-sub-df6ce9a7-7jhmw\" (UID: \"c3dd5471-c0c9-42c2-bed5-1edb497c0014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ce9a7-7jhmw" Oct 30 22:16:20.330949 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:20.330920 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ce9a7-7jhmw" Oct 30 22:16:20.715591 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:20.715557 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df6ce9a7-7jhmw"] Oct 30 22:16:20.717547 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:16:20.717515 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3dd5471_c0c9_42c2_bed5_1edb497c0014.slice/crio-21484919b4293a108511b7df4ef4005a92958bb3f722d88a517bc5cfc7fe0123 WatchSource:0}: Error finding container 21484919b4293a108511b7df4ef4005a92958bb3f722d88a517bc5cfc7fe0123: Status 404 returned error can't find the container with id 21484919b4293a108511b7df4ef4005a92958bb3f722d88a517bc5cfc7fe0123 Oct 30 22:16:21.307007 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:21.306959 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ce9a7-7jhmw" event={"ID":"c3dd5471-c0c9-42c2-bed5-1edb497c0014","Type":"ContainerStarted","Data":"2a4d2162df137a3f2718ded7fdc879ba568e2a57599a3c25316c8f0656a9bf64"} Oct 30 22:16:21.307489 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:21.307064 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ce9a7-7jhmw" event={"ID":"c3dd5471-c0c9-42c2-bed5-1edb497c0014","Type":"ContainerStarted","Data":"21484919b4293a108511b7df4ef4005a92958bb3f722d88a517bc5cfc7fe0123"} Oct 30 22:16:21.426665 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:21.426529 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ce9a7-7jhmw" podStartSLOduration=2.426508332 podStartE2EDuration="2.426508332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:21.419971564 +0000 UTC m=+1583.972147868" watchObservedRunningTime="2026-10-30 22:16:21.426508332 +0000 UTC m=+1583.978684633" Oct 30 22:16:22.318046 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:22.318005 6006 generic.go:356] "Generic (PLEG): container finished" podID="c3dd5471-c0c9-42c2-bed5-1edb497c0014" containerID="2a4d2162df137a3f2718ded7fdc879ba568e2a57599a3c25316c8f0656a9bf64" exitCode=0 Oct 30 22:16:22.318496 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:22.318092 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ce9a7-7jhmw" event={"ID":"c3dd5471-c0c9-42c2-bed5-1edb497c0014","Type":"ContainerDied","Data":"2a4d2162df137a3f2718ded7fdc879ba568e2a57599a3c25316c8f0656a9bf64"} Oct 30 22:16:23.530480 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:23.530444 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ce9a7-7jhmw" Oct 30 22:16:23.612288 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:23.612152 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3dd5471-c0c9-42c2-bed5-1edb497c0014-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3dd5471-c0c9-42c2-bed5-1edb497c0014-maas-api-openshift-ca\") pod \"c3dd5471-c0c9-42c2-bed5-1edb497c0014\" (UID: \"c3dd5471-c0c9-42c2-bed5-1edb497c0014\") " Oct 30 22:16:23.612288 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:23.612262 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3dd5471-c0c9-42c2-bed5-1edb497c0014-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3dd5471-c0c9-42c2-bed5-1edb497c0014-maas-api-xks-ca\") pod \"c3dd5471-c0c9-42c2-bed5-1edb497c0014\" (UID: \"c3dd5471-c0c9-42c2-bed5-1edb497c0014\") " Oct 30 22:16:23.612523 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:23.612331 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3dd5471-c0c9-42c2-bed5-1edb497c0014-kube-api-access-l4v8v\" (UniqueName: \"kubernetes.io/projected/c3dd5471-c0c9-42c2-bed5-1edb497c0014-kube-api-access-l4v8v\") pod \"c3dd5471-c0c9-42c2-bed5-1edb497c0014\" (UID: \"c3dd5471-c0c9-42c2-bed5-1edb497c0014\") " Oct 30 22:16:23.612637 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:23.612612 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3dd5471-c0c9-42c2-bed5-1edb497c0014-maas-api-openshift-ca" pod "c3dd5471-c0c9-42c2-bed5-1edb497c0014" (UID: "c3dd5471-c0c9-42c2-bed5-1edb497c0014"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:23.613499 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:23.613455 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3dd5471-c0c9-42c2-bed5-1edb497c0014-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:23.614763 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:23.614733 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3dd5471-c0c9-42c2-bed5-1edb497c0014-maas-api-xks-ca" pod "c3dd5471-c0c9-42c2-bed5-1edb497c0014" (UID: "c3dd5471-c0c9-42c2-bed5-1edb497c0014"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:23.614894 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:23.614864 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3dd5471-c0c9-42c2-bed5-1edb497c0014-kube-api-access-l4v8v" pod "c3dd5471-c0c9-42c2-bed5-1edb497c0014" (UID: "c3dd5471-c0c9-42c2-bed5-1edb497c0014"). InnerVolumeSpecName "kube-api-access-l4v8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:23.715132 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:23.715080 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3dd5471-c0c9-42c2-bed5-1edb497c0014-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:23.715132 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:23.715133 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4v8v\" (UniqueName: \"kubernetes.io/projected/c3dd5471-c0c9-42c2-bed5-1edb497c0014-kube-api-access-l4v8v\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:24.384273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:24.373157 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ce9a7-7jhmw" event={"ID":"c3dd5471-c0c9-42c2-bed5-1edb497c0014","Type":"ContainerDied","Data":"21484919b4293a108511b7df4ef4005a92958bb3f722d88a517bc5cfc7fe0123"} Oct 30 22:16:24.384273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:24.373226 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21484919b4293a108511b7df4ef4005a92958bb3f722d88a517bc5cfc7fe0123" Oct 30 22:16:24.384273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:24.373323 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6ce9a7-7jhmw" Oct 30 22:16:26.841275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:26.815610 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:26.914572 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:26.914523 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:27.354274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:27.349375 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:27.430277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:27.428422 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:27.521413 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:27.521363 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:27.550227 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:27.549350 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:27.588283 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:27.585351 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:16:27.588283 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:27.585422 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" podUID="500679e9-0b27-49c3-9b36-7fcfb7d1fc32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:27.821659 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:27.821613 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:27.823923 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:27.821966 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 22:16:27.824670 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:27.821989 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 22:16:27.828044 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:27.823195 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 22:16:27.828044 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:27.826874 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:16:27.828044 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:27.823232 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:27.921090 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:27.920442 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 22:16:27.921090 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:27.920981 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:16:28.116456 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:28.116288 6006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:28.118133 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:28.118088 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:16:28.122028 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:28.121994 6006 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:16:28.122277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:28.122230 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" podUID="46a061d3-8105-4260-9bf1-4fc3a5a09343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:16:28.159416 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:28.158822 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:16:28.212037 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:16:28.211663 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:16:30.860965 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:30.860908 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5513f6aa-4fcp9"] Oct 30 22:16:30.861802 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:30.861780 6006 cpu_manager.go:414] "containerMap: removing container" podUID="c3dd5471-c0c9-42c2-bed5-1edb497c0014" containerName="revoke-keys" Oct 30 22:16:30.861802 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:30.861803 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3dd5471-c0c9-42c2-bed5-1edb497c0014" containerName="revoke-keys" Oct 30 22:16:30.861940 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:30.861931 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3dd5471-c0c9-42c2-bed5-1edb497c0014" containerName="revoke-keys" Oct 30 22:16:30.866120 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:30.866098 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5513f6aa-4fcp9" Oct 30 22:16:30.911332 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:30.911288 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5d0d36c-96b0-4e99-b427-8bfe8a0528d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5513f6aa-4fcp9\" (UID: \"a5d0d36c-96b0-4e99-b427-8bfe8a0528d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5513f6aa-4fcp9" Oct 30 22:16:30.911736 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:30.911711 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh42k\" (UniqueName: \"kubernetes.io/projected/a5d0d36c-96b0-4e99-b427-8bfe8a0528d2-kube-api-access-sh42k\") pod \"maas-api-revoke-sub-5513f6aa-4fcp9\" (UID: \"a5d0d36c-96b0-4e99-b427-8bfe8a0528d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5513f6aa-4fcp9" Oct 30 22:16:30.911864 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:30.911763 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5d0d36c-96b0-4e99-b427-8bfe8a0528d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5513f6aa-4fcp9\" (UID: \"a5d0d36c-96b0-4e99-b427-8bfe8a0528d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5513f6aa-4fcp9" Oct 30 22:16:30.922165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:30.922133 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5513f6aa-4fcp9"] Oct 30 22:16:30.962391 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:30.962356 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:16:30.962584 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:30.962499 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:16:31.013983 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:31.013937 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh42k\" (UniqueName: \"kubernetes.io/projected/a5d0d36c-96b0-4e99-b427-8bfe8a0528d2-kube-api-access-sh42k\") pod \"maas-api-revoke-sub-5513f6aa-4fcp9\" (UID: \"a5d0d36c-96b0-4e99-b427-8bfe8a0528d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5513f6aa-4fcp9" Oct 30 22:16:31.014177 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:31.014008 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5d0d36c-96b0-4e99-b427-8bfe8a0528d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5513f6aa-4fcp9\" (UID: \"a5d0d36c-96b0-4e99-b427-8bfe8a0528d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5513f6aa-4fcp9" Oct 30 22:16:31.014319 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:31.014298 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5d0d36c-96b0-4e99-b427-8bfe8a0528d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5513f6aa-4fcp9\" (UID: \"a5d0d36c-96b0-4e99-b427-8bfe8a0528d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5513f6aa-4fcp9" Oct 30 22:16:31.015993 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:31.015964 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5d0d36c-96b0-4e99-b427-8bfe8a0528d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5513f6aa-4fcp9\" (UID: \"a5d0d36c-96b0-4e99-b427-8bfe8a0528d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5513f6aa-4fcp9" Oct 30 22:16:31.017593 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:31.017564 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5d0d36c-96b0-4e99-b427-8bfe8a0528d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5513f6aa-4fcp9\" (UID: \"a5d0d36c-96b0-4e99-b427-8bfe8a0528d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5513f6aa-4fcp9" Oct 30 22:16:31.073383 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:31.073348 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh42k\" (UniqueName: \"kubernetes.io/projected/a5d0d36c-96b0-4e99-b427-8bfe8a0528d2-kube-api-access-sh42k\") pod \"maas-api-revoke-sub-5513f6aa-4fcp9\" (UID: \"a5d0d36c-96b0-4e99-b427-8bfe8a0528d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5513f6aa-4fcp9" Oct 30 22:16:31.181416 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:31.181314 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5513f6aa-4fcp9" Oct 30 22:16:31.587570 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:16:31.587537 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d0d36c_96b0_4e99_b427_8bfe8a0528d2.slice/crio-eff6f05a563430f6105873be1208241117ca96a2a587f640654b1d31b37deb8b WatchSource:0}: Error finding container eff6f05a563430f6105873be1208241117ca96a2a587f640654b1d31b37deb8b: Status 404 returned error can't find the container with id eff6f05a563430f6105873be1208241117ca96a2a587f640654b1d31b37deb8b Oct 30 22:16:31.589444 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:31.589417 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5513f6aa-4fcp9"] Oct 30 22:16:31.622406 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:31.622370 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5513f6aa-4fcp9" event={"ID":"a5d0d36c-96b0-4e99-b427-8bfe8a0528d2","Type":"ContainerStarted","Data":"eff6f05a563430f6105873be1208241117ca96a2a587f640654b1d31b37deb8b"} Oct 30 22:16:32.632890 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:32.632769 6006 generic.go:356] "Generic (PLEG): container finished" podID="a5d0d36c-96b0-4e99-b427-8bfe8a0528d2" containerID="b5fdafc4a4243c7aab7c119da1922d338d25621dbb097b20eda97bbf92c02bc5" exitCode=0 Oct 30 22:16:32.633430 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:32.632838 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5513f6aa-4fcp9" event={"ID":"a5d0d36c-96b0-4e99-b427-8bfe8a0528d2","Type":"ContainerDied","Data":"b5fdafc4a4243c7aab7c119da1922d338d25621dbb097b20eda97bbf92c02bc5"} Oct 30 22:16:33.844763 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:33.844733 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5513f6aa-4fcp9" Oct 30 22:16:33.960007 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:33.959922 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5d0d36c-96b0-4e99-b427-8bfe8a0528d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5d0d36c-96b0-4e99-b427-8bfe8a0528d2-maas-api-openshift-ca\") pod \"a5d0d36c-96b0-4e99-b427-8bfe8a0528d2\" (UID: \"a5d0d36c-96b0-4e99-b427-8bfe8a0528d2\") " Oct 30 22:16:33.960386 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:33.960362 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5d0d36c-96b0-4e99-b427-8bfe8a0528d2-kube-api-access-sh42k\" (UniqueName: \"kubernetes.io/projected/a5d0d36c-96b0-4e99-b427-8bfe8a0528d2-kube-api-access-sh42k\") pod \"a5d0d36c-96b0-4e99-b427-8bfe8a0528d2\" (UID: \"a5d0d36c-96b0-4e99-b427-8bfe8a0528d2\") " Oct 30 22:16:33.960885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:33.960845 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5d0d36c-96b0-4e99-b427-8bfe8a0528d2-maas-api-openshift-ca" pod "a5d0d36c-96b0-4e99-b427-8bfe8a0528d2" (UID: "a5d0d36c-96b0-4e99-b427-8bfe8a0528d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:33.961022 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:33.960866 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5d0d36c-96b0-4e99-b427-8bfe8a0528d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5d0d36c-96b0-4e99-b427-8bfe8a0528d2-maas-api-xks-ca\") pod \"a5d0d36c-96b0-4e99-b427-8bfe8a0528d2\" (UID: \"a5d0d36c-96b0-4e99-b427-8bfe8a0528d2\") " Oct 30 22:16:33.963287 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:33.961708 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5d0d36c-96b0-4e99-b427-8bfe8a0528d2-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:33.965893 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:33.965864 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d0d36c-96b0-4e99-b427-8bfe8a0528d2-kube-api-access-sh42k" pod "a5d0d36c-96b0-4e99-b427-8bfe8a0528d2" (UID: "a5d0d36c-96b0-4e99-b427-8bfe8a0528d2"). InnerVolumeSpecName "kube-api-access-sh42k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:33.966417 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:33.966389 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5d0d36c-96b0-4e99-b427-8bfe8a0528d2-maas-api-xks-ca" pod "a5d0d36c-96b0-4e99-b427-8bfe8a0528d2" (UID: "a5d0d36c-96b0-4e99-b427-8bfe8a0528d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:34.062902 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:34.062861 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5d0d36c-96b0-4e99-b427-8bfe8a0528d2-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:34.062902 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:34.062910 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh42k\" (UniqueName: \"kubernetes.io/projected/a5d0d36c-96b0-4e99-b427-8bfe8a0528d2-kube-api-access-sh42k\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:34.651779 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:34.651740 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5513f6aa-4fcp9" Oct 30 22:16:34.651779 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:34.651756 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5513f6aa-4fcp9" event={"ID":"a5d0d36c-96b0-4e99-b427-8bfe8a0528d2","Type":"ContainerDied","Data":"eff6f05a563430f6105873be1208241117ca96a2a587f640654b1d31b37deb8b"} Oct 30 22:16:34.652002 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:34.651818 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eff6f05a563430f6105873be1208241117ca96a2a587f640654b1d31b37deb8b" Oct 30 22:16:47.962099 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:47.962059 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj"] Oct 30 22:16:47.962865 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:47.962845 6006 cpu_manager.go:414] "containerMap: removing container" podUID="a5d0d36c-96b0-4e99-b427-8bfe8a0528d2" containerName="revoke-keys" Oct 30 22:16:47.962918 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:47.962866 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5d0d36c-96b0-4e99-b427-8bfe8a0528d2" containerName="revoke-keys" Oct 30 22:16:47.962978 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:47.962969 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5d0d36c-96b0-4e99-b427-8bfe8a0528d2" containerName="revoke-keys" Oct 30 22:16:47.966823 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:47.966803 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:47.988808 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:47.988777 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-envoy\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:47.988979 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:47.988831 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-workload-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:47.988979 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:47.988940 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7acaef50-e202-45c0-ad2a-182925d51e0f-istiod-ca-cert\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:47.988979 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:47.988968 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-podinfo\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:47.989142 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:47.989119 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-data\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:47.989195 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:47.989156 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-credential-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:47.989335 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:47.989314 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-token\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:47.989387 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:47.989364 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-workload-certs\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:47.989475 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:47.989456 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5smz4\" (UniqueName: \"kubernetes.io/projected/7acaef50-e202-45c0-ad2a-182925d51e0f-kube-api-access-5smz4\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:48.075594 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:48.075548 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj"] Oct 30 22:16:48.090447 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:48.090409 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-data\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:48.090447 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:48.090451 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-credential-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:48.090679 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:48.090561 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-token\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:48.090679 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:48.090620 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-workload-certs\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:48.090785 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:48.090758 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5smz4\" (UniqueName: \"kubernetes.io/projected/7acaef50-e202-45c0-ad2a-182925d51e0f-kube-api-access-5smz4\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:48.090841 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:48.090829 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-envoy\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:48.090922 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:48.090891 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-workload-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:48.090979 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:48.090912 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-data\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:48.090979 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:48.090967 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-workload-certs\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:48.091084 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:48.090982 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7acaef50-e202-45c0-ad2a-182925d51e0f-istiod-ca-cert\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:48.091299 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:48.091234 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-workload-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:48.091444 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:48.091284 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-podinfo\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:48.091791 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:48.091763 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7acaef50-e202-45c0-ad2a-182925d51e0f-istiod-ca-cert\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:48.091952 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:48.091822 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-credential-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:48.093388 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:48.093362 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-envoy\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:48.094377 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:48.094354 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-podinfo\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:48.153185 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:48.153147 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-token\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:48.153405 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:48.153194 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5smz4\" (UniqueName: \"kubernetes.io/projected/7acaef50-e202-45c0-ad2a-182925d51e0f-kube-api-access-5smz4\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:48.282353 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:48.282228 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:48.721375 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:16:48.721324 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7acaef50_e202_45c0_ad2a_182925d51e0f.slice/crio-c17705a08f0c8c3ce99b1fcc62e26ddcaac293dbd6d0cca7eceadf4eab090f5d WatchSource:0}: Error finding container c17705a08f0c8c3ce99b1fcc62e26ddcaac293dbd6d0cca7eceadf4eab090f5d: Status 404 returned error can't find the container with id c17705a08f0c8c3ce99b1fcc62e26ddcaac293dbd6d0cca7eceadf4eab090f5d Oct 30 22:16:48.723958 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:48.723922 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj"] Oct 30 22:16:48.725361 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:48.725322 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:16:48.725470 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:48.725449 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:16:48.725518 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:48.725486 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:16:48.772877 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:48.772159 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" event={"ID":"7acaef50-e202-45c0-ad2a-182925d51e0f","Type":"ContainerStarted","Data":"c17705a08f0c8c3ce99b1fcc62e26ddcaac293dbd6d0cca7eceadf4eab090f5d"} Oct 30 22:16:49.782135 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:49.782086 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" event={"ID":"7acaef50-e202-45c0-ad2a-182925d51e0f","Type":"ContainerStarted","Data":"dc3ecd21dbcae16625544626920d900e86139e2f5c9f7a5da82e190c248ee51f"} Oct 30 22:16:49.953505 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:49.953431 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" podStartSLOduration=2.953411464 podStartE2EDuration="2.953411464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:49.945815753 +0000 UTC m=+1612.497992053" watchObservedRunningTime="2026-10-30 22:16:49.953411464 +0000 UTC m=+1612.505587764" Oct 30 22:16:50.282432 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:50.282398 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:50.284055 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:50.284023 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 22:16:50.284508 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:50.284079 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" podUID="7acaef50-e202-45c0-ad2a-182925d51e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 22:16:51.283081 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:51.283043 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 22:16:51.283519 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:51.283100 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" podUID="7acaef50-e202-45c0-ad2a-182925d51e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 22:16:52.283665 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:52.283625 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 22:16:52.284177 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:52.283691 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" podUID="7acaef50-e202-45c0-ad2a-182925d51e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 22:16:53.283124 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:53.283085 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 22:16:53.283348 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:53.283144 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" podUID="7acaef50-e202-45c0-ad2a-182925d51e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 22:16:54.131629 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:54.131592 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw"] Oct 30 22:16:54.137010 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:54.136963 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw" Oct 30 22:16:54.178297 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:54.178215 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw"] Oct 30 22:16:54.215670 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:54.199167 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a4df18-a649-4c24-b175-604ff0244581-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw\" (UID: \"94a4df18-a649-4c24-b175-604ff0244581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw" Oct 30 22:16:54.215670 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:54.199490 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s77g5\" (UniqueName: \"kubernetes.io/projected/94a4df18-a649-4c24-b175-604ff0244581-kube-api-access-s77g5\") pod \"maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw\" (UID: \"94a4df18-a649-4c24-b175-604ff0244581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw" Oct 30 22:16:54.215670 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:54.199525 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a4df18-a649-4c24-b175-604ff0244581-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw\" (UID: \"94a4df18-a649-4c24-b175-604ff0244581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw" Oct 30 22:16:54.227598 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:54.227559 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:16:54.229209 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:54.228455 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:16:54.283502 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:54.283469 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 22:16:54.283671 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:54.283522 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" podUID="7acaef50-e202-45c0-ad2a-182925d51e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 22:16:54.322277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:54.321416 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s77g5\" (UniqueName: \"kubernetes.io/projected/94a4df18-a649-4c24-b175-604ff0244581-kube-api-access-s77g5\") pod \"maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw\" (UID: \"94a4df18-a649-4c24-b175-604ff0244581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw" Oct 30 22:16:54.322277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:54.321477 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a4df18-a649-4c24-b175-604ff0244581-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw\" (UID: \"94a4df18-a649-4c24-b175-604ff0244581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw" Oct 30 22:16:54.322277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:54.321651 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a4df18-a649-4c24-b175-604ff0244581-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw\" (UID: \"94a4df18-a649-4c24-b175-604ff0244581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw" Oct 30 22:16:54.324024 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:54.323983 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a4df18-a649-4c24-b175-604ff0244581-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw\" (UID: \"94a4df18-a649-4c24-b175-604ff0244581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw" Oct 30 22:16:54.325990 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:54.325961 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a4df18-a649-4c24-b175-604ff0244581-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw\" (UID: \"94a4df18-a649-4c24-b175-604ff0244581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw" Oct 30 22:16:54.384425 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:54.384348 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s77g5\" (UniqueName: \"kubernetes.io/projected/94a4df18-a649-4c24-b175-604ff0244581-kube-api-access-s77g5\") pod \"maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw\" (UID: \"94a4df18-a649-4c24-b175-604ff0244581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw" Oct 30 22:16:54.454525 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:54.454439 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw" Oct 30 22:16:54.859315 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:54.859284 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw"] Oct 30 22:16:54.860887 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:16:54.860861 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a4df18_a649_4c24_b175_604ff0244581.slice/crio-6e8bec33b849f9d084d50e3fb58aff40230b1cd94f2770b971e0154610c4f210 WatchSource:0}: Error finding container 6e8bec33b849f9d084d50e3fb58aff40230b1cd94f2770b971e0154610c4f210: Status 404 returned error can't find the container with id 6e8bec33b849f9d084d50e3fb58aff40230b1cd94f2770b971e0154610c4f210 Oct 30 22:16:55.284039 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:55.283995 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 22:16:55.284498 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:55.284056 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" podUID="7acaef50-e202-45c0-ad2a-182925d51e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 22:16:55.876003 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:55.875954 6006 generic.go:356] "Generic (PLEG): container finished" podID="94a4df18-a649-4c24-b175-604ff0244581" containerID="79fbbc6b7c6adbe4084c0abaf349de6faccdc219f76bf75596048712f3a844d9" exitCode=0 Oct 30 22:16:55.876229 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:55.876042 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw" event={"ID":"94a4df18-a649-4c24-b175-604ff0244581","Type":"ContainerDied","Data":"79fbbc6b7c6adbe4084c0abaf349de6faccdc219f76bf75596048712f3a844d9"} Oct 30 22:16:55.876229 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:55.876100 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw" event={"ID":"94a4df18-a649-4c24-b175-604ff0244581","Type":"ContainerStarted","Data":"6e8bec33b849f9d084d50e3fb58aff40230b1cd94f2770b971e0154610c4f210"} Oct 30 22:16:56.287685 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:56.287650 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:56.288228 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:56.288208 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:56.289018 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:56.288996 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:16:57.063126 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:57.063100 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw" Oct 30 22:16:57.175689 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:57.175605 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94a4df18-a649-4c24-b175-604ff0244581-kube-api-access-s77g5\" (UniqueName: \"kubernetes.io/projected/94a4df18-a649-4c24-b175-604ff0244581-kube-api-access-s77g5\") pod \"94a4df18-a649-4c24-b175-604ff0244581\" (UID: \"94a4df18-a649-4c24-b175-604ff0244581\") " Oct 30 22:16:57.175879 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:57.175693 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94a4df18-a649-4c24-b175-604ff0244581-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a4df18-a649-4c24-b175-604ff0244581-maas-api-openshift-ca\") pod \"94a4df18-a649-4c24-b175-604ff0244581\" (UID: \"94a4df18-a649-4c24-b175-604ff0244581\") " Oct 30 22:16:57.175879 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:57.175741 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94a4df18-a649-4c24-b175-604ff0244581-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a4df18-a649-4c24-b175-604ff0244581-maas-api-xks-ca\") pod \"94a4df18-a649-4c24-b175-604ff0244581\" (UID: \"94a4df18-a649-4c24-b175-604ff0244581\") " Oct 30 22:16:57.176403 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:57.176091 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a4df18-a649-4c24-b175-604ff0244581-maas-api-openshift-ca" pod "94a4df18-a649-4c24-b175-604ff0244581" (UID: "94a4df18-a649-4c24-b175-604ff0244581"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:57.176588 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:57.176543 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a4df18-a649-4c24-b175-604ff0244581-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:57.179402 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:57.179372 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94a4df18-a649-4c24-b175-604ff0244581-maas-api-xks-ca" pod "94a4df18-a649-4c24-b175-604ff0244581" (UID: "94a4df18-a649-4c24-b175-604ff0244581"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:57.182307 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:57.182237 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a4df18-a649-4c24-b175-604ff0244581-kube-api-access-s77g5" pod "94a4df18-a649-4c24-b175-604ff0244581" (UID: "94a4df18-a649-4c24-b175-604ff0244581"). InnerVolumeSpecName "kube-api-access-s77g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:57.278322 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:57.278239 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s77g5\" (UniqueName: \"kubernetes.io/projected/94a4df18-a649-4c24-b175-604ff0244581-kube-api-access-s77g5\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:57.278322 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:57.278321 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a4df18-a649-4c24-b175-604ff0244581-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:57.898118 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:57.898032 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw" Oct 30 22:16:57.898539 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:57.898038 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw" event={"ID":"94a4df18-a649-4c24-b175-604ff0244581","Type":"ContainerDied","Data":"6e8bec33b849f9d084d50e3fb58aff40230b1cd94f2770b971e0154610c4f210"} Oct 30 22:16:57.898539 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:57.898177 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e8bec33b849f9d084d50e3fb58aff40230b1cd94f2770b971e0154610c4f210" Oct 30 22:16:57.914381 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:16:57.914338 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:16:58.788941 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:58.788898 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q"] Oct 30 22:16:58.789665 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:58.789637 6006 cpu_manager.go:414] "containerMap: removing container" podUID="94a4df18-a649-4c24-b175-604ff0244581" containerName="revoke-keys" Oct 30 22:16:58.789665 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:58.789658 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a4df18-a649-4c24-b175-604ff0244581" containerName="revoke-keys" Oct 30 22:16:58.789826 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:58.789797 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a4df18-a649-4c24-b175-604ff0244581" containerName="revoke-keys" Oct 30 22:16:58.793903 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:58.793880 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:58.895544 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:58.895503 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q"] Oct 30 22:16:58.907375 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:58.906425 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzr8n\" (UniqueName: \"kubernetes.io/projected/b246ab27-ac36-46ef-9db0-93545e5baefb-kube-api-access-tzr8n\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:58.907375 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:58.906520 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:58.907375 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:58.906638 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:58.907375 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:58.906780 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:58.907375 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:58.906819 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:58.907375 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:58.906860 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:58.907375 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:58.906910 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:58.907375 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:58.907003 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:58.907375 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:58.907133 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b246ab27-ac36-46ef-9db0-93545e5baefb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:59.009106 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:59.009063 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b246ab27-ac36-46ef-9db0-93545e5baefb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:59.009302 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:59.009256 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzr8n\" (UniqueName: \"kubernetes.io/projected/b246ab27-ac36-46ef-9db0-93545e5baefb-kube-api-access-tzr8n\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:59.009356 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:59.009313 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:59.009393 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:59.009353 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:59.009456 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:59.009439 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:59.009508 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:59.009478 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:59.009566 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:59.009522 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:59.009620 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:59.009570 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:59.009725 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:59.009674 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:59.009823 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:59.009792 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:59.009966 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:59.009929 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b246ab27-ac36-46ef-9db0-93545e5baefb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:59.010297 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:59.010270 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:59.010434 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:59.010344 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:59.010434 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:59.010351 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:59.012524 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:59.012501 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:59.016848 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:59.016821 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:59.068228 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:59.068192 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:59.068431 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:59.068402 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzr8n\" (UniqueName: \"kubernetes.io/projected/b246ab27-ac36-46ef-9db0-93545e5baefb-kube-api-access-tzr8n\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vnd7q\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:59.111964 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:59.111919 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:16:59.555332 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:16:59.555299 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb246ab27_ac36_46ef_9db0_93545e5baefb.slice/crio-0648ca28834cb9d8c62175798d839d04bc55b918c02e1ffb1b39c12899479e57 WatchSource:0}: Error finding container 0648ca28834cb9d8c62175798d839d04bc55b918c02e1ffb1b39c12899479e57: Status 404 returned error can't find the container with id 0648ca28834cb9d8c62175798d839d04bc55b918c02e1ffb1b39c12899479e57 Oct 30 22:16:59.557290 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:59.557223 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q"] Oct 30 22:16:59.558534 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:59.558492 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:16:59.558608 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:59.558578 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:16:59.558655 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:59.558622 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:16:59.922241 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:59.922142 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" event={"ID":"b246ab27-ac36-46ef-9db0-93545e5baefb","Type":"ContainerStarted","Data":"e36996a5a1250e25b244d009a07c9367ec7910e9829bb232d852b502e8876595"} Oct 30 22:16:59.922241 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:16:59.922208 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" event={"ID":"b246ab27-ac36-46ef-9db0-93545e5baefb","Type":"ContainerStarted","Data":"0648ca28834cb9d8c62175798d839d04bc55b918c02e1ffb1b39c12899479e57"} Oct 30 22:17:00.044381 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:00.044310 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" podStartSLOduration=2.044289858 podStartE2EDuration="2.044289858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:00.043552345 +0000 UTC m=+1622.595728645" watchObservedRunningTime="2026-10-30 22:17:00.044289858 +0000 UTC m=+1622.596466154" Oct 30 22:17:00.113184 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:00.113130 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:17:00.114937 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:00.114901 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-vnd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:17:00.115097 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:00.114958 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" podUID="b246ab27-ac36-46ef-9db0-93545e5baefb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 22:17:01.115274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:01.114740 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-vnd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:17:01.115274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:01.114807 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" podUID="b246ab27-ac36-46ef-9db0-93545e5baefb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 22:17:02.116273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:02.114695 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-vnd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:17:02.116273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:02.114765 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" podUID="b246ab27-ac36-46ef-9db0-93545e5baefb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 22:17:03.120276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:03.118724 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-vnd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:17:03.120276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:03.118796 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" podUID="b246ab27-ac36-46ef-9db0-93545e5baefb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 22:17:03.773481 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:03.769557 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn"] Oct 30 22:17:03.797303 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:03.787541 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn" Oct 30 22:17:03.856312 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:03.852006 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn"] Oct 30 22:17:03.873182 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:03.873145 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:17:03.876007 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:03.875952 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:17:03.915275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:03.896543 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/890d0569-13d7-4887-99c0-792b97831607-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn\" (UID: \"890d0569-13d7-4887-99c0-792b97831607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn" Oct 30 22:17:03.915275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:03.896624 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/890d0569-13d7-4887-99c0-792b97831607-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn\" (UID: \"890d0569-13d7-4887-99c0-792b97831607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn" Oct 30 22:17:03.915275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:03.896732 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5c6t\" (UniqueName: \"kubernetes.io/projected/890d0569-13d7-4887-99c0-792b97831607-kube-api-access-v5c6t\") pod \"maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn\" (UID: \"890d0569-13d7-4887-99c0-792b97831607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn" Oct 30 22:17:03.997751 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:03.997714 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/890d0569-13d7-4887-99c0-792b97831607-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn\" (UID: \"890d0569-13d7-4887-99c0-792b97831607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn" Oct 30 22:17:04.003964 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:04.003933 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/890d0569-13d7-4887-99c0-792b97831607-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn\" (UID: \"890d0569-13d7-4887-99c0-792b97831607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn" Oct 30 22:17:04.036488 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:04.003829 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/890d0569-13d7-4887-99c0-792b97831607-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn\" (UID: \"890d0569-13d7-4887-99c0-792b97831607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn" Oct 30 22:17:04.036488 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:04.036288 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/890d0569-13d7-4887-99c0-792b97831607-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn\" (UID: \"890d0569-13d7-4887-99c0-792b97831607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn" Oct 30 22:17:04.036722 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:04.036582 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5c6t\" (UniqueName: \"kubernetes.io/projected/890d0569-13d7-4887-99c0-792b97831607-kube-api-access-v5c6t\") pod \"maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn\" (UID: \"890d0569-13d7-4887-99c0-792b97831607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn" Oct 30 22:17:04.096232 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:04.096201 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5c6t\" (UniqueName: \"kubernetes.io/projected/890d0569-13d7-4887-99c0-792b97831607-kube-api-access-v5c6t\") pod \"maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn\" (UID: \"890d0569-13d7-4887-99c0-792b97831607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn" Oct 30 22:17:04.115354 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:04.113714 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-vnd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:17:04.115354 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:04.113782 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" podUID="b246ab27-ac36-46ef-9db0-93545e5baefb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 22:17:04.264317 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:04.264244 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn" Oct 30 22:17:04.667426 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:04.667388 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn"] Oct 30 22:17:04.667836 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:17:04.667804 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod890d0569_13d7_4887_99c0_792b97831607.slice/crio-bb83aef102527384e470df94e5710add1bf6c3c988dd534ed24037f6e1221306 WatchSource:0}: Error finding container bb83aef102527384e470df94e5710add1bf6c3c988dd534ed24037f6e1221306: Status 404 returned error can't find the container with id bb83aef102527384e470df94e5710add1bf6c3c988dd534ed24037f6e1221306 Oct 30 22:17:05.045093 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:05.045057 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn" event={"ID":"890d0569-13d7-4887-99c0-792b97831607","Type":"ContainerStarted","Data":"bb83aef102527384e470df94e5710add1bf6c3c988dd534ed24037f6e1221306"} Oct 30 22:17:05.117663 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:05.117624 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-vnd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:17:05.118085 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:05.118052 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" podUID="b246ab27-ac36-46ef-9db0-93545e5baefb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 22:17:06.055532 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:06.055495 6006 generic.go:356] "Generic (PLEG): container finished" podID="890d0569-13d7-4887-99c0-792b97831607" containerID="956f6bb2952fbf2b9213feb70ce88cc7e3e8b02dc667aa4844c51f441b2978b1" exitCode=0 Oct 30 22:17:06.055931 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:06.055563 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn" event={"ID":"890d0569-13d7-4887-99c0-792b97831607","Type":"ContainerDied","Data":"956f6bb2952fbf2b9213feb70ce88cc7e3e8b02dc667aa4844c51f441b2978b1"} Oct 30 22:17:06.113351 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:06.113312 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-vnd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:17:06.113553 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:06.113384 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" podUID="b246ab27-ac36-46ef-9db0-93545e5baefb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 22:17:07.127567 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:07.127529 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-vnd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:17:07.128193 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:07.128158 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" podUID="b246ab27-ac36-46ef-9db0-93545e5baefb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 22:17:07.373400 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:07.373372 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn" Oct 30 22:17:07.446445 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:07.446396 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/890d0569-13d7-4887-99c0-792b97831607-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/890d0569-13d7-4887-99c0-792b97831607-maas-api-openshift-ca\") pod \"890d0569-13d7-4887-99c0-792b97831607\" (UID: \"890d0569-13d7-4887-99c0-792b97831607\") " Oct 30 22:17:07.446625 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:07.446587 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/890d0569-13d7-4887-99c0-792b97831607-kube-api-access-v5c6t\" (UniqueName: \"kubernetes.io/projected/890d0569-13d7-4887-99c0-792b97831607-kube-api-access-v5c6t\") pod \"890d0569-13d7-4887-99c0-792b97831607\" (UID: \"890d0569-13d7-4887-99c0-792b97831607\") " Oct 30 22:17:07.446844 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:07.446827 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/890d0569-13d7-4887-99c0-792b97831607-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/890d0569-13d7-4887-99c0-792b97831607-maas-api-xks-ca\") pod \"890d0569-13d7-4887-99c0-792b97831607\" (UID: \"890d0569-13d7-4887-99c0-792b97831607\") " Oct 30 22:17:07.446926 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:07.446855 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/890d0569-13d7-4887-99c0-792b97831607-maas-api-openshift-ca" pod "890d0569-13d7-4887-99c0-792b97831607" (UID: "890d0569-13d7-4887-99c0-792b97831607"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:07.447493 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:07.447464 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/890d0569-13d7-4887-99c0-792b97831607-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:07.449024 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:07.448995 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/890d0569-13d7-4887-99c0-792b97831607-kube-api-access-v5c6t" pod "890d0569-13d7-4887-99c0-792b97831607" (UID: "890d0569-13d7-4887-99c0-792b97831607"). InnerVolumeSpecName "kube-api-access-v5c6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:07.449419 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:07.449394 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/890d0569-13d7-4887-99c0-792b97831607-maas-api-xks-ca" pod "890d0569-13d7-4887-99c0-792b97831607" (UID: "890d0569-13d7-4887-99c0-792b97831607"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:07.549109 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:07.549061 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/890d0569-13d7-4887-99c0-792b97831607-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:07.549109 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:07.549103 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5c6t\" (UniqueName: \"kubernetes.io/projected/890d0569-13d7-4887-99c0-792b97831607-kube-api-access-v5c6t\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:08.078104 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:08.078070 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn" Oct 30 22:17:08.078332 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:08.078103 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn" event={"ID":"890d0569-13d7-4887-99c0-792b97831607","Type":"ContainerDied","Data":"bb83aef102527384e470df94e5710add1bf6c3c988dd534ed24037f6e1221306"} Oct 30 22:17:08.078332 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:08.078165 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb83aef102527384e470df94e5710add1bf6c3c988dd534ed24037f6e1221306" Oct 30 22:17:08.512988 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:08.512888 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:17:08.513873 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:08.513849 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:17:08.514676 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:08.514657 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:17:09.080546 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:09.080514 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn"] Oct 30 22:17:09.081339 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:09.081317 6006 cpu_manager.go:414] "containerMap: removing container" podUID="890d0569-13d7-4887-99c0-792b97831607" containerName="revoke-keys" Oct 30 22:17:09.081339 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:09.081335 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="890d0569-13d7-4887-99c0-792b97831607" containerName="revoke-keys" Oct 30 22:17:09.081498 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:09.081488 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="890d0569-13d7-4887-99c0-792b97831607" containerName="revoke-keys" Oct 30 22:17:09.085509 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:09.085486 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn" Oct 30 22:17:09.135568 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:09.135539 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:17:09.139701 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:09.139646 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:17:09.143003 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:09.142976 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn"] Oct 30 22:17:09.177807 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:09.177764 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvx6j\" (UniqueName: \"kubernetes.io/projected/dfdf29a1-10fa-493f-946c-fb7569e3de16-kube-api-access-cvx6j\") pod \"maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn\" (UID: \"dfdf29a1-10fa-493f-946c-fb7569e3de16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn" Oct 30 22:17:09.177985 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:09.177931 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfdf29a1-10fa-493f-946c-fb7569e3de16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn\" (UID: \"dfdf29a1-10fa-493f-946c-fb7569e3de16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn" Oct 30 22:17:09.178259 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:09.178223 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfdf29a1-10fa-493f-946c-fb7569e3de16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn\" (UID: \"dfdf29a1-10fa-493f-946c-fb7569e3de16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn" Oct 30 22:17:09.279491 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:09.279454 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfdf29a1-10fa-493f-946c-fb7569e3de16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn\" (UID: \"dfdf29a1-10fa-493f-946c-fb7569e3de16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn" Oct 30 22:17:09.279651 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:09.279633 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvx6j\" (UniqueName: \"kubernetes.io/projected/dfdf29a1-10fa-493f-946c-fb7569e3de16-kube-api-access-cvx6j\") pod \"maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn\" (UID: \"dfdf29a1-10fa-493f-946c-fb7569e3de16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn" Oct 30 22:17:09.279735 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:09.279717 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfdf29a1-10fa-493f-946c-fb7569e3de16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn\" (UID: \"dfdf29a1-10fa-493f-946c-fb7569e3de16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn" Oct 30 22:17:09.280237 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:09.280210 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfdf29a1-10fa-493f-946c-fb7569e3de16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn\" (UID: \"dfdf29a1-10fa-493f-946c-fb7569e3de16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn" Oct 30 22:17:09.282572 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:09.282549 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfdf29a1-10fa-493f-946c-fb7569e3de16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn\" (UID: \"dfdf29a1-10fa-493f-946c-fb7569e3de16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn" Oct 30 22:17:09.338955 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:09.338876 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvx6j\" (UniqueName: \"kubernetes.io/projected/dfdf29a1-10fa-493f-946c-fb7569e3de16-kube-api-access-cvx6j\") pod \"maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn\" (UID: \"dfdf29a1-10fa-493f-946c-fb7569e3de16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn" Oct 30 22:17:09.406125 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:09.406091 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn" Oct 30 22:17:09.809313 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:17:09.808600 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfdf29a1_10fa_493f_946c_fb7569e3de16.slice/crio-37899c127b54824f207ee126fe14a5ea9d2d9e9871e096d9ae407e56d06d0901 WatchSource:0}: Error finding container 37899c127b54824f207ee126fe14a5ea9d2d9e9871e096d9ae407e56d06d0901: Status 404 returned error can't find the container with id 37899c127b54824f207ee126fe14a5ea9d2d9e9871e096d9ae407e56d06d0901 Oct 30 22:17:09.809313 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:09.808886 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn"] Oct 30 22:17:10.095468 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:10.095368 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn" event={"ID":"dfdf29a1-10fa-493f-946c-fb7569e3de16","Type":"ContainerStarted","Data":"37899c127b54824f207ee126fe14a5ea9d2d9e9871e096d9ae407e56d06d0901"} Oct 30 22:17:11.107150 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:11.107114 6006 generic.go:356] "Generic (PLEG): container finished" podID="dfdf29a1-10fa-493f-946c-fb7569e3de16" containerID="566d0ded391a908ff82aa3207a5115a6381963deec5820a8b3b14c96334a75e2" exitCode=0 Oct 30 22:17:11.107574 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:11.107204 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn" event={"ID":"dfdf29a1-10fa-493f-946c-fb7569e3de16","Type":"ContainerDied","Data":"566d0ded391a908ff82aa3207a5115a6381963deec5820a8b3b14c96334a75e2"} Oct 30 22:17:12.315993 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:12.315961 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn" Oct 30 22:17:12.449583 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:12.449473 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfdf29a1-10fa-493f-946c-fb7569e3de16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfdf29a1-10fa-493f-946c-fb7569e3de16-maas-api-xks-ca\") pod \"dfdf29a1-10fa-493f-946c-fb7569e3de16\" (UID: \"dfdf29a1-10fa-493f-946c-fb7569e3de16\") " Oct 30 22:17:12.449841 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:12.449814 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfdf29a1-10fa-493f-946c-fb7569e3de16-kube-api-access-cvx6j\" (UniqueName: \"kubernetes.io/projected/dfdf29a1-10fa-493f-946c-fb7569e3de16-kube-api-access-cvx6j\") pod \"dfdf29a1-10fa-493f-946c-fb7569e3de16\" (UID: \"dfdf29a1-10fa-493f-946c-fb7569e3de16\") " Oct 30 22:17:12.449956 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:12.449895 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfdf29a1-10fa-493f-946c-fb7569e3de16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfdf29a1-10fa-493f-946c-fb7569e3de16-maas-api-openshift-ca\") pod \"dfdf29a1-10fa-493f-946c-fb7569e3de16\" (UID: \"dfdf29a1-10fa-493f-946c-fb7569e3de16\") " Oct 30 22:17:12.450458 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:12.450432 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfdf29a1-10fa-493f-946c-fb7569e3de16-maas-api-openshift-ca" pod "dfdf29a1-10fa-493f-946c-fb7569e3de16" (UID: "dfdf29a1-10fa-493f-946c-fb7569e3de16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:12.452349 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:12.452222 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfdf29a1-10fa-493f-946c-fb7569e3de16-kube-api-access-cvx6j" pod "dfdf29a1-10fa-493f-946c-fb7569e3de16" (UID: "dfdf29a1-10fa-493f-946c-fb7569e3de16"). InnerVolumeSpecName "kube-api-access-cvx6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:12.452623 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:12.452592 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfdf29a1-10fa-493f-946c-fb7569e3de16-maas-api-xks-ca" pod "dfdf29a1-10fa-493f-946c-fb7569e3de16" (UID: "dfdf29a1-10fa-493f-946c-fb7569e3de16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:12.552144 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:12.552108 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvx6j\" (UniqueName: \"kubernetes.io/projected/dfdf29a1-10fa-493f-946c-fb7569e3de16-kube-api-access-cvx6j\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:12.552144 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:12.552139 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfdf29a1-10fa-493f-946c-fb7569e3de16-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:12.552144 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:12.552150 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfdf29a1-10fa-493f-946c-fb7569e3de16-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:13.128757 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:13.128724 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn" Oct 30 22:17:13.128956 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:13.128747 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn" event={"ID":"dfdf29a1-10fa-493f-946c-fb7569e3de16","Type":"ContainerDied","Data":"37899c127b54824f207ee126fe14a5ea9d2d9e9871e096d9ae407e56d06d0901"} Oct 30 22:17:13.128956 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:13.128809 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37899c127b54824f207ee126fe14a5ea9d2d9e9871e096d9ae407e56d06d0901" Oct 30 22:17:15.186280 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:15.186226 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:17:18.129651 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:18.129591 6006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:18.161396 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:18.161343 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:18.260511 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:18.260451 6006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Oct 30 22:17:18.345387 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:18.345337 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:18.417740 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:18.416377 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:18.443659 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:18.443585 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:27.956987 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:17:27.956937 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:17:43.284900 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:43.284861 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw"] Oct 30 22:17:43.331469 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:43.331434 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67184f2a-2rvdw"] Oct 30 22:17:43.607330 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:43.607292 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94a4df18-a649-4c24-b175-604ff0244581" path="/var/lib/kubelet/pods/94a4df18-a649-4c24-b175-604ff0244581/volumes" Oct 30 22:17:50.244944 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:50.244877 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn"] Oct 30 22:17:50.289464 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:50.288350 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ff716ef6-j2wjn"] Oct 30 22:17:51.606801 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:51.606752 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="890d0569-13d7-4887-99c0-792b97831607" path="/var/lib/kubelet/pods/890d0569-13d7-4887-99c0-792b97831607/volumes" Oct 30 22:17:51.709075 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:51.709039 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn"] Oct 30 22:17:51.758770 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:51.758736 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-72b5472b-mpzvn"] Oct 30 22:17:53.621521 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:17:53.621476 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfdf29a1-10fa-493f-946c-fb7569e3de16" path="/var/lib/kubelet/pods/dfdf29a1-10fa-493f-946c-fb7569e3de16/volumes" Oct 30 22:17:57.924575 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:17:57.924507 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:18:27.912569 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:18:27.912517 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:18:30.050708 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:18:30.050671 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f16663a3-htn9q"] Oct 30 22:18:30.097163 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:18:30.097125 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f16663a3-htn9q"] Oct 30 22:18:31.607309 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:18:31.607238 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68086ae0-79c5-403b-bebb-03ae08256b41" path="/var/lib/kubelet/pods/68086ae0-79c5-403b-bebb-03ae08256b41/volumes" Oct 30 22:18:35.067632 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:18:35.067595 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88af5db5-v5rds"] Oct 30 22:18:35.088500 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:18:35.088412 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88af5db5-v5rds"] Oct 30 22:18:35.605513 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:18:35.605473 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b03b6525-1963-4875-91d9-025d1e884994" path="/var/lib/kubelet/pods/b03b6525-1963-4875-91d9-025d1e884994/volumes" Oct 30 22:18:48.745949 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:18:48.745902 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:18:57.914999 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:18:57.914947 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:19:06.107361 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:06.107305 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bfd8ad6-4zqlc"] Oct 30 22:19:06.113588 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:06.113551 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bfd8ad6-4zqlc"] Oct 30 22:19:07.606915 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:07.606872 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="775d1260-3327-48a0-a7f6-47b62265e310" path="/var/lib/kubelet/pods/775d1260-3327-48a0-a7f6-47b62265e310/volumes" Oct 30 22:19:17.054862 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:17.054821 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-702f477e-xtz54"] Oct 30 22:19:17.102224 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:17.102183 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-702f477e-xtz54"] Oct 30 22:19:17.608163 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:17.608121 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12332804-45e1-4613-b6ed-1e0f7ca91b65" path="/var/lib/kubelet/pods/12332804-45e1-4613-b6ed-1e0f7ca91b65/volumes" Oct 30 22:19:27.063731 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:27.063692 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9de8e643-zn27k"] Oct 30 22:19:27.112493 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:27.112453 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9de8e643-zn27k"] Oct 30 22:19:27.606657 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:27.606618 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20468fa7-f3f8-46d1-a05e-6744f4ada9e6" path="/var/lib/kubelet/pods/20468fa7-f3f8-46d1-a05e-6744f4ada9e6/volumes" Oct 30 22:19:27.917514 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:19:27.917414 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:19:33.057301 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:33.057225 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a709f15-bcvzh"] Oct 30 22:19:33.106720 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:33.106673 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a709f15-bcvzh"] Oct 30 22:19:33.607403 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:33.607357 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f5e21fa-7968-45c9-a8d8-dc086e155f65" path="/var/lib/kubelet/pods/8f5e21fa-7968-45c9-a8d8-dc086e155f65/volumes" Oct 30 22:19:36.857717 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:36.857680 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:19:37.514429 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:37.501972 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb"] Oct 30 22:19:37.657818 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:37.657782 6006 cpu_manager.go:414] "containerMap: removing container" podUID="dfdf29a1-10fa-493f-946c-fb7569e3de16" containerName="revoke-keys" Oct 30 22:19:37.658054 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:37.658039 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfdf29a1-10fa-493f-946c-fb7569e3de16" containerName="revoke-keys" Oct 30 22:19:37.658665 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:37.658504 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfdf29a1-10fa-493f-946c-fb7569e3de16" containerName="revoke-keys" Oct 30 22:19:37.688319 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:37.686704 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb"] Oct 30 22:19:37.688499 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:37.688470 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb" Oct 30 22:19:37.757224 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:37.757184 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ba51dc-4019-42e6-9063-b981dde3b757-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb\" (UID: \"21ba51dc-4019-42e6-9063-b981dde3b757\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb" Oct 30 22:19:37.757451 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:37.757300 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42sj6\" (UniqueName: \"kubernetes.io/projected/21ba51dc-4019-42e6-9063-b981dde3b757-kube-api-access-42sj6\") pod \"maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb\" (UID: \"21ba51dc-4019-42e6-9063-b981dde3b757\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb" Oct 30 22:19:37.757543 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:37.757518 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ba51dc-4019-42e6-9063-b981dde3b757-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb\" (UID: \"21ba51dc-4019-42e6-9063-b981dde3b757\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb" Oct 30 22:19:37.786560 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:37.786464 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:19:37.786560 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:37.786469 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:19:37.859155 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:37.859117 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ba51dc-4019-42e6-9063-b981dde3b757-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb\" (UID: \"21ba51dc-4019-42e6-9063-b981dde3b757\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb" Oct 30 22:19:37.859727 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:37.859230 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42sj6\" (UniqueName: \"kubernetes.io/projected/21ba51dc-4019-42e6-9063-b981dde3b757-kube-api-access-42sj6\") pod \"maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb\" (UID: \"21ba51dc-4019-42e6-9063-b981dde3b757\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb" Oct 30 22:19:37.859727 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:37.859383 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ba51dc-4019-42e6-9063-b981dde3b757-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb\" (UID: \"21ba51dc-4019-42e6-9063-b981dde3b757\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb" Oct 30 22:19:37.860133 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:37.860111 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ba51dc-4019-42e6-9063-b981dde3b757-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb\" (UID: \"21ba51dc-4019-42e6-9063-b981dde3b757\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb" Oct 30 22:19:37.862720 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:37.862691 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ba51dc-4019-42e6-9063-b981dde3b757-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb\" (UID: \"21ba51dc-4019-42e6-9063-b981dde3b757\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb" Oct 30 22:19:37.919365 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:37.919334 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42sj6\" (UniqueName: \"kubernetes.io/projected/21ba51dc-4019-42e6-9063-b981dde3b757-kube-api-access-42sj6\") pod \"maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb\" (UID: \"21ba51dc-4019-42e6-9063-b981dde3b757\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb" Oct 30 22:19:38.009423 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:38.009382 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb" Oct 30 22:19:38.609794 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:19:38.609752 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ba51dc_4019_42e6_9063_b981dde3b757.slice/crio-c4b1011b4d4a3e87b9307e92609394665fc8269f27db428c93411d8f1443d3c8 WatchSource:0}: Error finding container c4b1011b4d4a3e87b9307e92609394665fc8269f27db428c93411d8f1443d3c8: Status 404 returned error can't find the container with id c4b1011b4d4a3e87b9307e92609394665fc8269f27db428c93411d8f1443d3c8 Oct 30 22:19:38.611648 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:38.611620 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb"] Oct 30 22:19:39.034623 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.034572 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb" event={"ID":"21ba51dc-4019-42e6-9063-b981dde3b757","Type":"ContainerStarted","Data":"e2d41ffab8c7590f1a30ca186ebac03140c2bcc3a86b1a34c66fd131145f11b0"} Oct 30 22:19:39.035094 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.034646 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb" event={"ID":"21ba51dc-4019-42e6-9063-b981dde3b757","Type":"ContainerStarted","Data":"c4b1011b4d4a3e87b9307e92609394665fc8269f27db428c93411d8f1443d3c8"} Oct 30 22:19:39.181302 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.181211 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb" podStartSLOduration=2.181193067 podStartE2EDuration="2.181193067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:39.176492584 +0000 UTC m=+1781.728668884" watchObservedRunningTime="2026-10-30 22:19:39.181193067 +0000 UTC m=+1781.733369368" Oct 30 22:19:39.402169 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.402133 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5"] Oct 30 22:19:39.406743 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.406712 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.509277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.507422 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-ed44da-openshift-default-dockercfg-9jtjj\"" Oct 30 22:19:39.533308 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.532091 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5"] Oct 30 22:19:39.620278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.619222 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-data\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.620278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.619529 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-credential-socket\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.620278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.619573 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-envoy\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.620278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.619606 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-podinfo\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.620278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.619683 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkh8k\" (UniqueName: \"kubernetes.io/projected/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-kube-api-access-nkh8k\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.620278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.619715 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-workload-socket\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.620278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.619747 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istiod-ca-cert\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.620278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.619791 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-workload-certs\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.620278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.619823 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-token\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.741329 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.725180 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-credential-socket\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.741329 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.725266 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-envoy\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.741329 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.725302 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-podinfo\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.741329 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.725380 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkh8k\" (UniqueName: \"kubernetes.io/projected/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-kube-api-access-nkh8k\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.741329 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.725412 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-workload-socket\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.741329 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.725445 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istiod-ca-cert\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.741329 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.725493 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-workload-certs\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.741329 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.725524 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-token\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.741329 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.725650 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-data\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.741329 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.726192 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-data\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.741329 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.728144 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-workload-socket\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.804410 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.782750 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-credential-socket\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.804410 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.783599 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-podinfo\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.804410 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.801456 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istiod-ca-cert\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.852989 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.831693 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-workload-certs\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.871932 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.871894 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-token\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.880588 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.879992 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkh8k\" (UniqueName: \"kubernetes.io/projected/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-kube-api-access-nkh8k\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:39.890449 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:39.886501 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-envoy\") pod \"e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:40.030013 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:40.029906 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:40.058639 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:40.058596 6006 generic.go:356] "Generic (PLEG): container finished" podID="21ba51dc-4019-42e6-9063-b981dde3b757" containerID="e2d41ffab8c7590f1a30ca186ebac03140c2bcc3a86b1a34c66fd131145f11b0" exitCode=0 Oct 30 22:19:40.059104 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:40.058815 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb" event={"ID":"21ba51dc-4019-42e6-9063-b981dde3b757","Type":"ContainerDied","Data":"e2d41ffab8c7590f1a30ca186ebac03140c2bcc3a86b1a34c66fd131145f11b0"} Oct 30 22:19:40.523685 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:19:40.523648 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa2bdf13_4ade_4f01_8a3c_7463d62efd49.slice/crio-90a1390c1964160424a539987111ff29a6be62557175fd4a01e15c87ee16273b WatchSource:0}: Error finding container 90a1390c1964160424a539987111ff29a6be62557175fd4a01e15c87ee16273b: Status 404 returned error can't find the container with id 90a1390c1964160424a539987111ff29a6be62557175fd4a01e15c87ee16273b Oct 30 22:19:40.524723 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:40.524681 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5"] Oct 30 22:19:40.526780 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:40.526739 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:19:40.526893 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:40.526825 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:19:40.526951 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:40.526896 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:19:41.071635 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:41.071596 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" event={"ID":"aa2bdf13-4ade-4f01-8a3c-7463d62efd49","Type":"ContainerStarted","Data":"9e870379876ff473f4bf057ce3a7611f8d90365794d7086c38c4e32d4d5b3aa4"} Oct 30 22:19:41.072146 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:41.071661 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" event={"ID":"aa2bdf13-4ade-4f01-8a3c-7463d62efd49","Type":"ContainerStarted","Data":"90a1390c1964160424a539987111ff29a6be62557175fd4a01e15c87ee16273b"} Oct 30 22:19:41.193934 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:41.193867 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" podStartSLOduration=2.193849576 podStartE2EDuration="2.193849576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:41.192095119 +0000 UTC m=+1783.744271420" watchObservedRunningTime="2026-10-30 22:19:41.193849576 +0000 UTC m=+1783.746025876" Oct 30 22:19:41.314649 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:41.314614 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb" Oct 30 22:19:41.423405 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:41.421985 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21ba51dc-4019-42e6-9063-b981dde3b757-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ba51dc-4019-42e6-9063-b981dde3b757-maas-api-xks-ca\") pod \"21ba51dc-4019-42e6-9063-b981dde3b757\" (UID: \"21ba51dc-4019-42e6-9063-b981dde3b757\") " Oct 30 22:19:41.423631 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:41.423539 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21ba51dc-4019-42e6-9063-b981dde3b757-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ba51dc-4019-42e6-9063-b981dde3b757-maas-api-openshift-ca\") pod \"21ba51dc-4019-42e6-9063-b981dde3b757\" (UID: \"21ba51dc-4019-42e6-9063-b981dde3b757\") " Oct 30 22:19:41.423965 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:41.423936 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ba51dc-4019-42e6-9063-b981dde3b757-maas-api-openshift-ca" pod "21ba51dc-4019-42e6-9063-b981dde3b757" (UID: "21ba51dc-4019-42e6-9063-b981dde3b757"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:41.424261 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:41.424226 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ba51dc-4019-42e6-9063-b981dde3b757-kube-api-access-42sj6\" (UniqueName: \"kubernetes.io/projected/21ba51dc-4019-42e6-9063-b981dde3b757-kube-api-access-42sj6\") pod \"21ba51dc-4019-42e6-9063-b981dde3b757\" (UID: \"21ba51dc-4019-42e6-9063-b981dde3b757\") " Oct 30 22:19:41.425262 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:41.425224 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ba51dc-4019-42e6-9063-b981dde3b757-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:41.428862 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:41.428819 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21ba51dc-4019-42e6-9063-b981dde3b757-maas-api-xks-ca" pod "21ba51dc-4019-42e6-9063-b981dde3b757" (UID: "21ba51dc-4019-42e6-9063-b981dde3b757"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:19:41.429174 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:41.429137 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ba51dc-4019-42e6-9063-b981dde3b757-kube-api-access-42sj6" pod "21ba51dc-4019-42e6-9063-b981dde3b757" (UID: "21ba51dc-4019-42e6-9063-b981dde3b757"). InnerVolumeSpecName "kube-api-access-42sj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:41.529300 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:41.527335 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42sj6\" (UniqueName: \"kubernetes.io/projected/21ba51dc-4019-42e6-9063-b981dde3b757-kube-api-access-42sj6\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:41.529300 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:41.527385 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ba51dc-4019-42e6-9063-b981dde3b757-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:41.893589 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:41.893550 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk"] Oct 30 22:19:41.894354 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:41.894331 6006 cpu_manager.go:414] "containerMap: removing container" podUID="21ba51dc-4019-42e6-9063-b981dde3b757" containerName="revoke-keys" Oct 30 22:19:41.894354 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:41.894355 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ba51dc-4019-42e6-9063-b981dde3b757" containerName="revoke-keys" Oct 30 22:19:41.894599 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:41.894559 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ba51dc-4019-42e6-9063-b981dde3b757" containerName="revoke-keys" Oct 30 22:19:41.898224 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:41.898176 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk" Oct 30 22:19:41.936271 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:41.930537 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk"] Oct 30 22:19:41.949025 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:41.948991 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4da07739\"" Oct 30 22:19:41.949387 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:41.949366 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4da07739-dockercfg-zczwl\"" Oct 30 22:19:41.984219 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:41.984051 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6srx5\" (UniqueName: \"kubernetes.io/projected/7ddedf10-57e9-42d9-b1e7-3be877aba9d4-kube-api-access-6srx5\") pod \"payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk\" (UID: \"7ddedf10-57e9-42d9-b1e7-3be877aba9d4\") " pod="openshift-ingress/payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk" Oct 30 22:19:41.984751 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:41.984711 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ddedf10-57e9-42d9-b1e7-3be877aba9d4-plugins-config-volume\") pod \"payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk\" (UID: \"7ddedf10-57e9-42d9-b1e7-3be877aba9d4\") " pod="openshift-ingress/payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk" Oct 30 22:19:42.030597 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:42.030557 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:42.039449 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:42.039411 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:42.085431 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:42.085371 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb" Oct 30 22:19:42.087437 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:42.086237 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb" event={"ID":"21ba51dc-4019-42e6-9063-b981dde3b757","Type":"ContainerDied","Data":"c4b1011b4d4a3e87b9307e92609394665fc8269f27db428c93411d8f1443d3c8"} Oct 30 22:19:42.087437 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:42.086663 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4b1011b4d4a3e87b9307e92609394665fc8269f27db428c93411d8f1443d3c8" Oct 30 22:19:42.087437 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:42.086689 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:42.087916 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:42.087875 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:19:42.088111 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:42.088083 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ddedf10-57e9-42d9-b1e7-3be877aba9d4-plugins-config-volume\") pod \"payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk\" (UID: \"7ddedf10-57e9-42d9-b1e7-3be877aba9d4\") " pod="openshift-ingress/payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk" Oct 30 22:19:42.089098 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:42.089040 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ddedf10-57e9-42d9-b1e7-3be877aba9d4-plugins-config-volume\") pod \"payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk\" (UID: \"7ddedf10-57e9-42d9-b1e7-3be877aba9d4\") " pod="openshift-ingress/payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk" Oct 30 22:19:42.090511 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:42.090485 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6srx5\" (UniqueName: \"kubernetes.io/projected/7ddedf10-57e9-42d9-b1e7-3be877aba9d4-kube-api-access-6srx5\") pod \"payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk\" (UID: \"7ddedf10-57e9-42d9-b1e7-3be877aba9d4\") " pod="openshift-ingress/payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk" Oct 30 22:19:42.149910 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:42.149869 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6srx5\" (UniqueName: \"kubernetes.io/projected/7ddedf10-57e9-42d9-b1e7-3be877aba9d4-kube-api-access-6srx5\") pod \"payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk\" (UID: \"7ddedf10-57e9-42d9-b1e7-3be877aba9d4\") " pod="openshift-ingress/payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk" Oct 30 22:19:42.241324 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:42.235565 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk" Oct 30 22:19:42.686645 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:42.686613 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk"] Oct 30 22:19:42.687426 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:19:42.687393 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ddedf10_57e9_42d9_b1e7_3be877aba9d4.slice/crio-f932559dcdf9f55511e74032619f4defba5b8ef4d0c17b7272a3d1fb1080f810 WatchSource:0}: Error finding container f932559dcdf9f55511e74032619f4defba5b8ef4d0c17b7272a3d1fb1080f810: Status 404 returned error can't find the container with id f932559dcdf9f55511e74032619f4defba5b8ef4d0c17b7272a3d1fb1080f810 Oct 30 22:19:43.095939 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:43.095899 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk" event={"ID":"7ddedf10-57e9-42d9-b1e7-3be877aba9d4","Type":"ContainerStarted","Data":"29dbcd9bdf8413a1e1fa4f053a60542a79aa9a7c1cfe594b0a9c09eefa2e5adb"} Oct 30 22:19:43.096526 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:43.096002 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk" event={"ID":"7ddedf10-57e9-42d9-b1e7-3be877aba9d4","Type":"ContainerStarted","Data":"f932559dcdf9f55511e74032619f4defba5b8ef4d0c17b7272a3d1fb1080f810"} Oct 30 22:19:43.096526 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:43.096171 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk" Oct 30 22:19:43.206729 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:43.206663 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk" podStartSLOduration=2.206646542 podStartE2EDuration="2.206646542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:43.206243808 +0000 UTC m=+1785.758420111" watchObservedRunningTime="2026-10-30 22:19:43.206646542 +0000 UTC m=+1785.758822843" Oct 30 22:19:45.077702 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:45.077662 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9db548fb-k89sh"] Oct 30 22:19:45.123590 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:45.123544 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9db548fb-k89sh"] Oct 30 22:19:45.606846 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:45.606802 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="605fdfa0-1deb-4036-94c9-7f02f947245b" path="/var/lib/kubelet/pods/605fdfa0-1deb-4036-94c9-7f02f947245b/volumes" Oct 30 22:19:48.735674 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:48.735639 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7"] Oct 30 22:19:48.740366 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:48.740337 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7" Oct 30 22:19:48.759345 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:48.759225 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5txcr\" (UniqueName: \"kubernetes.io/projected/5c5e3d88-1dfc-476c-8f88-bf35edbbc82a-kube-api-access-5txcr\") pod \"payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7\" (UID: \"5c5e3d88-1dfc-476c-8f88-bf35edbbc82a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7" Oct 30 22:19:48.759548 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:48.759527 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5e3d88-1dfc-476c-8f88-bf35edbbc82a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7\" (UID: \"5c5e3d88-1dfc-476c-8f88-bf35edbbc82a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7" Oct 30 22:19:48.841512 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:48.841479 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-ed44da\"" Oct 30 22:19:48.845457 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:48.845384 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-ed44da-dockercfg-g9rcq\"" Oct 30 22:19:48.873275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:48.866403 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5e3d88-1dfc-476c-8f88-bf35edbbc82a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7\" (UID: \"5c5e3d88-1dfc-476c-8f88-bf35edbbc82a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7" Oct 30 22:19:48.873275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:48.866666 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5txcr\" (UniqueName: \"kubernetes.io/projected/5c5e3d88-1dfc-476c-8f88-bf35edbbc82a-kube-api-access-5txcr\") pod \"payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7\" (UID: \"5c5e3d88-1dfc-476c-8f88-bf35edbbc82a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7" Oct 30 22:19:48.873275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:48.867927 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5e3d88-1dfc-476c-8f88-bf35edbbc82a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7\" (UID: \"5c5e3d88-1dfc-476c-8f88-bf35edbbc82a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7" Oct 30 22:19:48.873275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:48.867985 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7"] Oct 30 22:19:49.001310 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:49.001095 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m"] Oct 30 22:19:49.011704 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:49.011672 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m"] Oct 30 22:19:49.011891 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:49.011818 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m" Oct 30 22:19:49.013115 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:49.013082 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5txcr\" (UniqueName: \"kubernetes.io/projected/5c5e3d88-1dfc-476c-8f88-bf35edbbc82a-kube-api-access-5txcr\") pod \"payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7\" (UID: \"5c5e3d88-1dfc-476c-8f88-bf35edbbc82a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7" Oct 30 22:19:49.058535 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:49.058500 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7" Oct 30 22:19:49.124828 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:49.122215 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3dc9f55-2232-45df-bff5-e307125462de-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m\" (UID: \"d3dc9f55-2232-45df-bff5-e307125462de\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m" Oct 30 22:19:49.124828 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:49.124488 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftt4v\" (UniqueName: \"kubernetes.io/projected/d3dc9f55-2232-45df-bff5-e307125462de-kube-api-access-ftt4v\") pod \"payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m\" (UID: \"d3dc9f55-2232-45df-bff5-e307125462de\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m" Oct 30 22:19:49.230321 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:49.229398 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftt4v\" (UniqueName: \"kubernetes.io/projected/d3dc9f55-2232-45df-bff5-e307125462de-kube-api-access-ftt4v\") pod \"payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m\" (UID: \"d3dc9f55-2232-45df-bff5-e307125462de\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m" Oct 30 22:19:49.230321 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:49.229688 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3dc9f55-2232-45df-bff5-e307125462de-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m\" (UID: \"d3dc9f55-2232-45df-bff5-e307125462de\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m" Oct 30 22:19:49.230855 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:49.230828 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3dc9f55-2232-45df-bff5-e307125462de-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m\" (UID: \"d3dc9f55-2232-45df-bff5-e307125462de\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m" Oct 30 22:19:49.304700 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:49.304662 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftt4v\" (UniqueName: \"kubernetes.io/projected/d3dc9f55-2232-45df-bff5-e307125462de-kube-api-access-ftt4v\") pod \"payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m\" (UID: \"d3dc9f55-2232-45df-bff5-e307125462de\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m" Oct 30 22:19:49.338804 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:49.338519 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m" Oct 30 22:19:49.575631 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:49.575601 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7"] Oct 30 22:19:49.577566 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:19:49.577533 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5e3d88_1dfc_476c_8f88_bf35edbbc82a.slice/crio-8563f27ef41364f6968813e2ed2ed74ca37b45c6acfd61ba6e50c038d3a56f14 WatchSource:0}: Error finding container 8563f27ef41364f6968813e2ed2ed74ca37b45c6acfd61ba6e50c038d3a56f14: Status 404 returned error can't find the container with id 8563f27ef41364f6968813e2ed2ed74ca37b45c6acfd61ba6e50c038d3a56f14 Oct 30 22:19:49.766613 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:49.766588 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m"] Oct 30 22:19:49.768809 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:19:49.768778 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3dc9f55_2232_45df_bff5_e307125462de.slice/crio-1750d60d18e14776f5d71003e808f51ab632b340e7fa7d4ab4976d964cd4dc13 WatchSource:0}: Error finding container 1750d60d18e14776f5d71003e808f51ab632b340e7fa7d4ab4976d964cd4dc13: Status 404 returned error can't find the container with id 1750d60d18e14776f5d71003e808f51ab632b340e7fa7d4ab4976d964cd4dc13 Oct 30 22:19:50.203968 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:50.203871 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7" event={"ID":"5c5e3d88-1dfc-476c-8f88-bf35edbbc82a","Type":"ContainerStarted","Data":"2bf51d6752d6c96206dd0c92f4570ef609c8a0179a6e43db92d59fcd51b76b35"} Oct 30 22:19:50.204163 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:50.203980 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7" Oct 30 22:19:50.204163 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:50.204016 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7" event={"ID":"5c5e3d88-1dfc-476c-8f88-bf35edbbc82a","Type":"ContainerStarted","Data":"8563f27ef41364f6968813e2ed2ed74ca37b45c6acfd61ba6e50c038d3a56f14"} Oct 30 22:19:50.205848 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:50.205821 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m" event={"ID":"d3dc9f55-2232-45df-bff5-e307125462de","Type":"ContainerStarted","Data":"dd9a95e92a563940496750a563e39a72678794494f34c35ac20280865a88f525"} Oct 30 22:19:50.205996 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:50.205868 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m" event={"ID":"d3dc9f55-2232-45df-bff5-e307125462de","Type":"ContainerStarted","Data":"1750d60d18e14776f5d71003e808f51ab632b340e7fa7d4ab4976d964cd4dc13"} Oct 30 22:19:50.205996 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:50.205905 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m" Oct 30 22:19:50.317893 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:50.317831 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7" podStartSLOduration=2.317813978 podStartE2EDuration="2.317813978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:50.313745535 +0000 UTC m=+1792.865921835" watchObservedRunningTime="2026-10-30 22:19:50.317813978 +0000 UTC m=+1792.869990281" Oct 30 22:19:50.450271 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:50.440899 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m" podStartSLOduration=2.440877584 podStartE2EDuration="2.440877584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:50.424581456 +0000 UTC m=+1792.976757755" watchObservedRunningTime="2026-10-30 22:19:50.440877584 +0000 UTC m=+1792.993053887" Oct 30 22:19:51.174658 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:51.174624 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bee2db53-tbhss"] Oct 30 22:19:51.181383 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:51.181353 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bee2db53-tbhss" Oct 30 22:19:51.233123 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:51.233070 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:19:51.275261 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:51.275210 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-248m4\" (UniqueName: \"kubernetes.io/projected/181c0684-588d-44ed-b0de-8122c234cc64-kube-api-access-248m4\") pod \"maas-api-revoke-keys-e2e-derive-bee2db53-tbhss\" (UID: \"181c0684-588d-44ed-b0de-8122c234cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bee2db53-tbhss" Oct 30 22:19:51.275456 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:51.275291 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/181c0684-588d-44ed-b0de-8122c234cc64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-bee2db53-tbhss\" (UID: \"181c0684-588d-44ed-b0de-8122c234cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bee2db53-tbhss" Oct 30 22:19:51.275588 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:51.275560 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/181c0684-588d-44ed-b0de-8122c234cc64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-bee2db53-tbhss\" (UID: \"181c0684-588d-44ed-b0de-8122c234cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bee2db53-tbhss" Oct 30 22:19:51.288577 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:51.279967 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:19:51.288577 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:51.287166 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bee2db53-tbhss"] Oct 30 22:19:51.377786 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:51.377722 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-248m4\" (UniqueName: \"kubernetes.io/projected/181c0684-588d-44ed-b0de-8122c234cc64-kube-api-access-248m4\") pod \"maas-api-revoke-keys-e2e-derive-bee2db53-tbhss\" (UID: \"181c0684-588d-44ed-b0de-8122c234cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bee2db53-tbhss" Oct 30 22:19:51.377999 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:51.377812 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/181c0684-588d-44ed-b0de-8122c234cc64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-bee2db53-tbhss\" (UID: \"181c0684-588d-44ed-b0de-8122c234cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bee2db53-tbhss" Oct 30 22:19:51.378086 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:51.378014 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/181c0684-588d-44ed-b0de-8122c234cc64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-bee2db53-tbhss\" (UID: \"181c0684-588d-44ed-b0de-8122c234cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bee2db53-tbhss" Oct 30 22:19:51.381853 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:51.379028 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/181c0684-588d-44ed-b0de-8122c234cc64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-bee2db53-tbhss\" (UID: \"181c0684-588d-44ed-b0de-8122c234cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bee2db53-tbhss" Oct 30 22:19:51.407041 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:51.407010 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/181c0684-588d-44ed-b0de-8122c234cc64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-bee2db53-tbhss\" (UID: \"181c0684-588d-44ed-b0de-8122c234cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bee2db53-tbhss" Oct 30 22:19:51.438073 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:51.438002 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-248m4\" (UniqueName: \"kubernetes.io/projected/181c0684-588d-44ed-b0de-8122c234cc64-kube-api-access-248m4\") pod \"maas-api-revoke-keys-e2e-derive-bee2db53-tbhss\" (UID: \"181c0684-588d-44ed-b0de-8122c234cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bee2db53-tbhss" Oct 30 22:19:51.497939 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:51.497901 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bee2db53-tbhss" Oct 30 22:19:51.893435 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:19:51.893405 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181c0684_588d_44ed_b0de_8122c234cc64.slice/crio-002b355aa339578e43c69ededaf9f3d86c100fdca95a27e369324e17fc607c2c WatchSource:0}: Error finding container 002b355aa339578e43c69ededaf9f3d86c100fdca95a27e369324e17fc607c2c: Status 404 returned error can't find the container with id 002b355aa339578e43c69ededaf9f3d86c100fdca95a27e369324e17fc607c2c Oct 30 22:19:51.893848 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:51.893823 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bee2db53-tbhss"] Oct 30 22:19:52.227479 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:52.227444 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bee2db53-tbhss" event={"ID":"181c0684-588d-44ed-b0de-8122c234cc64","Type":"ContainerStarted","Data":"002b355aa339578e43c69ededaf9f3d86c100fdca95a27e369324e17fc607c2c"} Oct 30 22:19:53.242052 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:53.242008 6006 generic.go:356] "Generic (PLEG): container finished" podID="181c0684-588d-44ed-b0de-8122c234cc64" containerID="50285152caa67245309d0bdfcb823765466773d2db1ecc30fab513689a28a054" exitCode=0 Oct 30 22:19:53.245489 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:53.242152 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bee2db53-tbhss" event={"ID":"181c0684-588d-44ed-b0de-8122c234cc64","Type":"ContainerDied","Data":"50285152caa67245309d0bdfcb823765466773d2db1ecc30fab513689a28a054"} Oct 30 22:19:54.110361 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:54.110328 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk" Oct 30 22:19:54.478165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:54.478134 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bee2db53-tbhss" Oct 30 22:19:54.639976 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:54.639933 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/181c0684-588d-44ed-b0de-8122c234cc64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/181c0684-588d-44ed-b0de-8122c234cc64-maas-api-openshift-ca\") pod \"181c0684-588d-44ed-b0de-8122c234cc64\" (UID: \"181c0684-588d-44ed-b0de-8122c234cc64\") " Oct 30 22:19:54.640205 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:54.640077 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/181c0684-588d-44ed-b0de-8122c234cc64-kube-api-access-248m4\" (UniqueName: \"kubernetes.io/projected/181c0684-588d-44ed-b0de-8122c234cc64-kube-api-access-248m4\") pod \"181c0684-588d-44ed-b0de-8122c234cc64\" (UID: \"181c0684-588d-44ed-b0de-8122c234cc64\") " Oct 30 22:19:54.640550 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:54.640516 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181c0684-588d-44ed-b0de-8122c234cc64-maas-api-openshift-ca" pod "181c0684-588d-44ed-b0de-8122c234cc64" (UID: "181c0684-588d-44ed-b0de-8122c234cc64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:54.640749 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:54.640721 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/181c0684-588d-44ed-b0de-8122c234cc64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/181c0684-588d-44ed-b0de-8122c234cc64-maas-api-xks-ca\") pod \"181c0684-588d-44ed-b0de-8122c234cc64\" (UID: \"181c0684-588d-44ed-b0de-8122c234cc64\") " Oct 30 22:19:54.642563 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:54.642538 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/181c0684-588d-44ed-b0de-8122c234cc64-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:54.643266 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:54.643217 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181c0684-588d-44ed-b0de-8122c234cc64-kube-api-access-248m4" pod "181c0684-588d-44ed-b0de-8122c234cc64" (UID: "181c0684-588d-44ed-b0de-8122c234cc64"). InnerVolumeSpecName "kube-api-access-248m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:54.650276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:54.644665 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/181c0684-588d-44ed-b0de-8122c234cc64-maas-api-xks-ca" pod "181c0684-588d-44ed-b0de-8122c234cc64" (UID: "181c0684-588d-44ed-b0de-8122c234cc64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:19:54.745341 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:54.745222 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-248m4\" (UniqueName: \"kubernetes.io/projected/181c0684-588d-44ed-b0de-8122c234cc64-kube-api-access-248m4\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:54.745341 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:54.745304 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/181c0684-588d-44ed-b0de-8122c234cc64-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:55.262210 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:55.262175 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bee2db53-tbhss" Oct 30 22:19:55.262210 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:55.262189 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bee2db53-tbhss" event={"ID":"181c0684-588d-44ed-b0de-8122c234cc64","Type":"ContainerDied","Data":"002b355aa339578e43c69ededaf9f3d86c100fdca95a27e369324e17fc607c2c"} Oct 30 22:19:55.262481 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:55.262335 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="002b355aa339578e43c69ededaf9f3d86c100fdca95a27e369324e17fc607c2c" Oct 30 22:19:57.922926 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:19:57.922886 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:19:57.946562 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:57.946535 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 22:19:57.961748 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:57.961723 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 22:19:58.202825 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:58.202713 6006 scope.go:122] "RemoveContainer" containerID="13e37090876081c28c8b32773cfc61058e3608618f856670b3189a0d24ab6798" Oct 30 22:19:58.217165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:58.217139 6006 scope.go:122] "RemoveContainer" containerID="97f58a684ec1023b33ee7a4cb96c88be853a67528d5bbe977655a93f983a3e06" Oct 30 22:19:59.050180 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:59.050146 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c960764-7f6x7"] Oct 30 22:19:59.095283 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:59.095239 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c960764-7f6x7"] Oct 30 22:19:59.606846 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:19:59.606810 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ee862a8-ab50-4538-8f6c-67ff8396afae" path="/var/lib/kubelet/pods/3ee862a8-ab50-4538-8f6c-67ff8396afae/volumes" Oct 30 22:20:01.219238 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:20:01.219204 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m" Oct 30 22:20:01.219786 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:20:01.219309 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7" Oct 30 22:20:24.428543 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:20:24.428508 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb"] Oct 30 22:20:24.474765 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:20:24.474726 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5024ad-hmkfb"] Oct 30 22:20:25.605662 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:20:25.605627 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21ba51dc-4019-42e6-9063-b981dde3b757" path="/var/lib/kubelet/pods/21ba51dc-4019-42e6-9063-b981dde3b757/volumes" Oct 30 22:20:27.980981 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:20:27.980940 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:20:40.791643 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:20:40.791601 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bee2db53-tbhss"] Oct 30 22:20:40.862065 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:20:40.862025 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bee2db53-tbhss"] Oct 30 22:20:41.612753 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:20:41.612713 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="181c0684-588d-44ed-b0de-8122c234cc64" path="/var/lib/kubelet/pods/181c0684-588d-44ed-b0de-8122c234cc64/volumes" Oct 30 22:20:42.012351 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:20:42.012221 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:20:56.055437 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:20:56.055400 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-374a479d-gnb7w"] Oct 30 22:20:56.108183 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:20:56.108141 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-374a479d-gnb7w"] Oct 30 22:20:57.606864 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:20:57.606830 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3e047c1-c154-4169-b4b4-79849a7f8f1e" path="/var/lib/kubelet/pods/c3e047c1-c154-4169-b4b4-79849a7f8f1e/volumes" Oct 30 22:20:57.913723 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:20:57.913624 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:20:58.258019 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:20:58.257934 6006 scope.go:122] "RemoveContainer" containerID="37c8d8e1c0aced2dda20c6ce9032ced00b5048bf2b5543cc1aa839489e313d26" Oct 30 22:20:58.268875 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:20:58.268848 6006 scope.go:122] "RemoveContainer" containerID="453f616f3eecb2d406d359b37ab843ccceded783d6881bb12d4a7f51c594f0f8" Oct 30 22:20:58.278885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:20:58.278862 6006 scope.go:122] "RemoveContainer" containerID="a4ab22e2ab892ab5f7b26a7d02d39557a9bfffba9608422bb89fae132b28e721" Oct 30 22:20:58.289737 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:20:58.289716 6006 scope.go:122] "RemoveContainer" containerID="9fb642f492efab9aec2d2f4584cf984455008c272d641daf398bfa02eb47c1a6" Oct 30 22:20:58.301149 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:20:58.301120 6006 scope.go:122] "RemoveContainer" containerID="372ce842e1fed14f9149049ffb270c3570069964e6f7f939d246fbf6110f8a09" Oct 30 22:20:58.313985 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:20:58.313955 6006 scope.go:122] "RemoveContainer" containerID="f318b884d9a900f895d4ddecd1d98ffcf6bf63cecbdaa53c4d1d489872a39d93" Oct 30 22:21:13.053717 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:13.053676 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e126c47b-t6vw7"] Oct 30 22:21:13.104304 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:13.104267 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e126c47b-t6vw7"] Oct 30 22:21:13.606510 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:13.606476 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a160973-4fcc-4efa-ba15-fe94f2a8cbd8" path="/var/lib/kubelet/pods/8a160973-4fcc-4efa-ba15-fe94f2a8cbd8/volumes" Oct 30 22:21:20.185735 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:20.185694 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt"] Oct 30 22:21:20.186217 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:20.185974 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" podUID="34b11b2f-2eb6-4313-9a00-74be3c58fecd" containerName="istio-proxy" containerID="cri-o://926b17f945f88db77ff307ac8264332a147db9020a96ccb8eb7508ca3bafa007" gracePeriod=30 Oct 30 22:21:20.239007 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:20.238970 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw"] Oct 30 22:21:20.239206 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:20.239184 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" podUID="d4a885f9-5fd5-4782-978b-ceb9fb740610" containerName="istio-proxy" containerID="cri-o://0c659a0ec05b303725b1b8ba27a92dfc351c1d382bc21a171eb29baf8eca6e2f" gracePeriod=30 Oct 30 22:21:20.244965 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:20.244934 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm"] Oct 30 22:21:20.245238 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:20.245189 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" podUID="7cce6f74-3f9a-49cf-a594-211f3bb1f432" containerName="istio-proxy" containerID="cri-o://69e726c5872b7976fdc0fef0721a30048e24a2ddef9f040cd58341f3e86d8fb8" gracePeriod=30 Oct 30 22:21:20.289347 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:20.289311 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942"] Oct 30 22:21:20.289588 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:20.289562 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" podUID="bdbec1e1-b705-464b-88bd-6b6ddbfb3513" containerName="istio-proxy" containerID="cri-o://94858957c1fa3efe4e033dd3c7eb1043af3f042f9e6a331f60c29b01339c6d9b" gracePeriod=30 Oct 30 22:21:23.148337 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:23.148295 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:21:23.148841 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:23.148386 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" podUID="34b11b2f-2eb6-4313-9a00-74be3c58fecd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:21:24.053724 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:24.053688 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df6ce9a7-7jhmw"] Oct 30 22:21:24.106741 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:24.106707 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df6ce9a7-7jhmw"] Oct 30 22:21:24.837975 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:24.837937 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:21:24.838463 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:24.838012 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" podUID="d4a885f9-5fd5-4782-978b-ceb9fb740610" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:21:25.288981 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.288947 6006 patch_prober.go:29] interesting pod/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:21:25.289124 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.289003 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" podUID="bdbec1e1-b705-464b-88bd-6b6ddbfb3513" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:21:25.609649 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.609608 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3dd5471-c0c9-42c2-bed5-1edb497c0014" path="/var/lib/kubelet/pods/c3dd5471-c0c9-42c2-bed5-1edb497c0014/volumes" Oct 30 22:21:25.614910 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.614474 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:21:25.679976 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.679946 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-envoy\") pod \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " Oct 30 22:21:25.680184 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.680000 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-token\" (UniqueName: \"kubernetes.io/projected/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-token\") pod \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " Oct 30 22:21:25.680184 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.680026 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-data\") pod \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " Oct 30 22:21:25.680184 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.680069 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b11b2f-2eb6-4313-9a00-74be3c58fecd-kube-api-access-hg2n8\" (UniqueName: \"kubernetes.io/projected/34b11b2f-2eb6-4313-9a00-74be3c58fecd-kube-api-access-hg2n8\") pod \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " Oct 30 22:21:25.680184 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.680139 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-podinfo\") pod \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " Oct 30 22:21:25.680403 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.680226 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-workload-socket\") pod \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " Oct 30 22:21:25.680403 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.680311 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istiod-ca-cert\") pod \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " Oct 30 22:21:25.680565 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.680529 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-credential-socket\") pod \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " Oct 30 22:21:25.680743 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.680647 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istiod-ca-cert" pod "34b11b2f-2eb6-4313-9a00-74be3c58fecd" (UID: "34b11b2f-2eb6-4313-9a00-74be3c58fecd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:25.680743 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.680669 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-workload-socket" pod "34b11b2f-2eb6-4313-9a00-74be3c58fecd" (UID: "34b11b2f-2eb6-4313-9a00-74be3c58fecd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:25.680743 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.680707 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-workload-certs\") pod \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\" (UID: \"34b11b2f-2eb6-4313-9a00-74be3c58fecd\") " Oct 30 22:21:25.681019 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.680998 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-data" pod "34b11b2f-2eb6-4313-9a00-74be3c58fecd" (UID: "34b11b2f-2eb6-4313-9a00-74be3c58fecd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:25.681105 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.681028 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-workload-certs" pod "34b11b2f-2eb6-4313-9a00-74be3c58fecd" (UID: "34b11b2f-2eb6-4313-9a00-74be3c58fecd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:25.681105 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.681060 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-credential-socket" pod "34b11b2f-2eb6-4313-9a00-74be3c58fecd" (UID: "34b11b2f-2eb6-4313-9a00-74be3c58fecd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:25.681686 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.681659 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.681793 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.681690 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.681793 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.681708 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.681793 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.681723 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.681793 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.681736 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.683998 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.683971 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b11b2f-2eb6-4313-9a00-74be3c58fecd-kube-api-access-hg2n8" pod "34b11b2f-2eb6-4313-9a00-74be3c58fecd" (UID: "34b11b2f-2eb6-4313-9a00-74be3c58fecd"). InnerVolumeSpecName "kube-api-access-hg2n8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:25.685056 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.685025 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-envoy" pod "34b11b2f-2eb6-4313-9a00-74be3c58fecd" (UID: "34b11b2f-2eb6-4313-9a00-74be3c58fecd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:25.685151 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.685103 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-podinfo" pod "34b11b2f-2eb6-4313-9a00-74be3c58fecd" (UID: "34b11b2f-2eb6-4313-9a00-74be3c58fecd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:21:25.685363 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.685336 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-token" pod "34b11b2f-2eb6-4313-9a00-74be3c58fecd" (UID: "34b11b2f-2eb6-4313-9a00-74be3c58fecd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:25.723816 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.722920 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:21:25.732002 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.731967 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:21:25.736165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.736144 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:21:25.782986 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.782954 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-workload-certs\") pod \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " Oct 30 22:21:25.783170 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783012 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-token\" (UniqueName: \"kubernetes.io/projected/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-token\") pod \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " Oct 30 22:21:25.783170 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783059 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-token\" (UniqueName: \"kubernetes.io/projected/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-token\") pod \"d4a885f9-5fd5-4782-978b-ceb9fb740610\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " Oct 30 22:21:25.783170 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783082 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-credential-socket\") pod \"d4a885f9-5fd5-4782-978b-ceb9fb740610\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " Oct 30 22:21:25.783170 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783118 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-workload-socket\") pod \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " Oct 30 22:21:25.783170 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783159 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-data\") pod \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " Oct 30 22:21:25.783475 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783205 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-data\") pod \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " Oct 30 22:21:25.783475 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783287 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-workload-socket\") pod \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " Oct 30 22:21:25.783475 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783317 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-workload-certs" pod "7cce6f74-3f9a-49cf-a594-211f3bb1f432" (UID: "7cce6f74-3f9a-49cf-a594-211f3bb1f432"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:25.783475 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783366 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-credential-socket" pod "d4a885f9-5fd5-4782-978b-ceb9fb740610" (UID: "d4a885f9-5fd5-4782-978b-ceb9fb740610"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:25.783475 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783393 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-token\" (UniqueName: \"kubernetes.io/projected/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-token\") pod \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " Oct 30 22:21:25.783475 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783452 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-envoy\") pod \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " Oct 30 22:21:25.783781 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783481 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istiod-ca-cert\") pod \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " Oct 30 22:21:25.783781 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783489 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-data" pod "7cce6f74-3f9a-49cf-a594-211f3bb1f432" (UID: "7cce6f74-3f9a-49cf-a594-211f3bb1f432"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:25.783781 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783575 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-kube-api-access-7cm9b\" (UniqueName: \"kubernetes.io/projected/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-kube-api-access-7cm9b\") pod \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " Oct 30 22:21:25.783781 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783581 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-workload-socket" pod "7cce6f74-3f9a-49cf-a594-211f3bb1f432" (UID: "7cce6f74-3f9a-49cf-a594-211f3bb1f432"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:25.783781 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783588 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-data" pod "bdbec1e1-b705-464b-88bd-6b6ddbfb3513" (UID: "bdbec1e1-b705-464b-88bd-6b6ddbfb3513"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:25.783781 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783602 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-workload-socket" pod "bdbec1e1-b705-464b-88bd-6b6ddbfb3513" (UID: "bdbec1e1-b705-464b-88bd-6b6ddbfb3513"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:25.783781 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783625 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-envoy\") pod \"d4a885f9-5fd5-4782-978b-ceb9fb740610\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " Oct 30 22:21:25.783781 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783668 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cce6f74-3f9a-49cf-a594-211f3bb1f432-kube-api-access-bxr85\" (UniqueName: \"kubernetes.io/projected/7cce6f74-3f9a-49cf-a594-211f3bb1f432-kube-api-access-bxr85\") pod \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " Oct 30 22:21:25.783781 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783701 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-envoy\") pod \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " Oct 30 22:21:25.783781 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783735 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-podinfo\") pod \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " Oct 30 22:21:25.783781 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783759 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-podinfo\") pod \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " Oct 30 22:21:25.784482 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783796 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4a885f9-5fd5-4782-978b-ceb9fb740610-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4a885f9-5fd5-4782-978b-ceb9fb740610-istiod-ca-cert\") pod \"d4a885f9-5fd5-4782-978b-ceb9fb740610\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " Oct 30 22:21:25.784482 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783822 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-credential-socket\") pod \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\" (UID: \"7cce6f74-3f9a-49cf-a594-211f3bb1f432\") " Oct 30 22:21:25.784482 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783880 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4a885f9-5fd5-4782-978b-ceb9fb740610-kube-api-access-2n9m7\" (UniqueName: \"kubernetes.io/projected/d4a885f9-5fd5-4782-978b-ceb9fb740610-kube-api-access-2n9m7\") pod \"d4a885f9-5fd5-4782-978b-ceb9fb740610\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " Oct 30 22:21:25.784482 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783918 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-credential-socket\") pod \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " Oct 30 22:21:25.784482 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783947 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istiod-ca-cert" pod "7cce6f74-3f9a-49cf-a594-211f3bb1f432" (UID: "7cce6f74-3f9a-49cf-a594-211f3bb1f432"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:25.784482 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.783987 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-podinfo\") pod \"d4a885f9-5fd5-4782-978b-ceb9fb740610\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " Oct 30 22:21:25.784482 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.784049 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-workload-socket\") pod \"d4a885f9-5fd5-4782-978b-ceb9fb740610\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " Oct 30 22:21:25.784482 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.784319 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4a885f9-5fd5-4782-978b-ceb9fb740610-istiod-ca-cert" pod "d4a885f9-5fd5-4782-978b-ceb9fb740610" (UID: "d4a885f9-5fd5-4782-978b-ceb9fb740610"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:25.784900 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.784821 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-credential-socket" pod "7cce6f74-3f9a-49cf-a594-211f3bb1f432" (UID: "7cce6f74-3f9a-49cf-a594-211f3bb1f432"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:25.784900 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.784850 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-workload-socket" pod "d4a885f9-5fd5-4782-978b-ceb9fb740610" (UID: "d4a885f9-5fd5-4782-978b-ceb9fb740610"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:25.786140 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.785027 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-credential-socket" pod "bdbec1e1-b705-464b-88bd-6b6ddbfb3513" (UID: "bdbec1e1-b705-464b-88bd-6b6ddbfb3513"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:25.786140 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.785215 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istiod-ca-cert\") pod \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " Oct 30 22:21:25.786140 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.785318 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-workload-certs\") pod \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\" (UID: \"bdbec1e1-b705-464b-88bd-6b6ddbfb3513\") " Oct 30 22:21:25.786140 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.785350 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-workload-certs\") pod \"d4a885f9-5fd5-4782-978b-ceb9fb740610\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " Oct 30 22:21:25.786140 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.785435 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-data\") pod \"d4a885f9-5fd5-4782-978b-ceb9fb740610\" (UID: \"d4a885f9-5fd5-4782-978b-ceb9fb740610\") " Oct 30 22:21:25.786140 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.785696 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-workload-certs" pod "d4a885f9-5fd5-4782-978b-ceb9fb740610" (UID: "d4a885f9-5fd5-4782-978b-ceb9fb740610"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:25.786140 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.785819 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-workload-certs" pod "bdbec1e1-b705-464b-88bd-6b6ddbfb3513" (UID: "bdbec1e1-b705-464b-88bd-6b6ddbfb3513"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:25.786140 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.785988 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-data" pod "d4a885f9-5fd5-4782-978b-ceb9fb740610" (UID: "d4a885f9-5fd5-4782-978b-ceb9fb740610"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:25.786140 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.786056 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istiod-ca-cert" pod "bdbec1e1-b705-464b-88bd-6b6ddbfb3513" (UID: "bdbec1e1-b705-464b-88bd-6b6ddbfb3513"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:25.786959 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.786934 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.787050 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.786966 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4a885f9-5fd5-4782-978b-ceb9fb740610-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.787050 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.786982 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.787050 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.787000 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.787050 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.787014 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.787050 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.787029 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.787050 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.787043 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.787398 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.787057 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.787398 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.787073 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.787398 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.787088 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.787398 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.787102 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.787398 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.787116 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.787398 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.787131 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg2n8\" (UniqueName: \"kubernetes.io/projected/34b11b2f-2eb6-4313-9a00-74be3c58fecd-kube-api-access-hg2n8\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.787398 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.787146 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.787398 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.787160 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.787398 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.787173 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b11b2f-2eb6-4313-9a00-74be3c58fecd-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.787398 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.787188 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.787398 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.787201 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.787398 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.787215 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.788023 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.787943 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-token" pod "d4a885f9-5fd5-4782-978b-ceb9fb740610" (UID: "d4a885f9-5fd5-4782-978b-ceb9fb740610"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:25.788160 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.788112 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-token" pod "7cce6f74-3f9a-49cf-a594-211f3bb1f432" (UID: "7cce6f74-3f9a-49cf-a594-211f3bb1f432"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:25.788238 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.788179 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-envoy" pod "bdbec1e1-b705-464b-88bd-6b6ddbfb3513" (UID: "bdbec1e1-b705-464b-88bd-6b6ddbfb3513"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:25.788414 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.788389 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-token" pod "bdbec1e1-b705-464b-88bd-6b6ddbfb3513" (UID: "bdbec1e1-b705-464b-88bd-6b6ddbfb3513"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:25.788613 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.788590 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4a885f9-5fd5-4782-978b-ceb9fb740610-kube-api-access-2n9m7" pod "d4a885f9-5fd5-4782-978b-ceb9fb740610" (UID: "d4a885f9-5fd5-4782-978b-ceb9fb740610"). InnerVolumeSpecName "kube-api-access-2n9m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:25.788701 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.788636 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-podinfo" pod "bdbec1e1-b705-464b-88bd-6b6ddbfb3513" (UID: "bdbec1e1-b705-464b-88bd-6b6ddbfb3513"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:21:25.788849 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.788815 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-kube-api-access-7cm9b" pod "bdbec1e1-b705-464b-88bd-6b6ddbfb3513" (UID: "bdbec1e1-b705-464b-88bd-6b6ddbfb3513"). InnerVolumeSpecName "kube-api-access-7cm9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:25.788917 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.788853 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-podinfo" pod "7cce6f74-3f9a-49cf-a594-211f3bb1f432" (UID: "7cce6f74-3f9a-49cf-a594-211f3bb1f432"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:21:25.789031 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.788994 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cce6f74-3f9a-49cf-a594-211f3bb1f432-kube-api-access-bxr85" pod "7cce6f74-3f9a-49cf-a594-211f3bb1f432" (UID: "7cce6f74-3f9a-49cf-a594-211f3bb1f432"). InnerVolumeSpecName "kube-api-access-bxr85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:25.789099 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.789053 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-envoy" pod "7cce6f74-3f9a-49cf-a594-211f3bb1f432" (UID: "7cce6f74-3f9a-49cf-a594-211f3bb1f432"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:25.789142 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.789125 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-envoy" pod "d4a885f9-5fd5-4782-978b-ceb9fb740610" (UID: "d4a885f9-5fd5-4782-978b-ceb9fb740610"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:25.789517 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.789478 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-podinfo" pod "d4a885f9-5fd5-4782-978b-ceb9fb740610" (UID: "d4a885f9-5fd5-4782-978b-ceb9fb740610"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:21:25.888237 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.888199 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.888237 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.888232 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxr85\" (UniqueName: \"kubernetes.io/projected/7cce6f74-3f9a-49cf-a594-211f3bb1f432-kube-api-access-bxr85\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.888701 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.888244 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.888701 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.888284 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.888701 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.888293 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.888701 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.888302 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n9m7\" (UniqueName: \"kubernetes.io/projected/d4a885f9-5fd5-4782-978b-ceb9fb740610-kube-api-access-2n9m7\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.888701 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.888311 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.888701 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.888319 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.888701 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.888330 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4a885f9-5fd5-4782-978b-ceb9fb740610-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.888701 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.888338 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.888701 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.888346 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cce6f74-3f9a-49cf-a594-211f3bb1f432-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:25.888701 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:25.888355 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cm9b\" (UniqueName: \"kubernetes.io/projected/bdbec1e1-b705-464b-88bd-6b6ddbfb3513-kube-api-access-7cm9b\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:26.193880 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.193843 6006 generic.go:356] "Generic (PLEG): container finished" podID="d4a885f9-5fd5-4782-978b-ceb9fb740610" containerID="0c659a0ec05b303725b1b8ba27a92dfc351c1d382bc21a171eb29baf8eca6e2f" exitCode=0 Oct 30 22:21:26.194074 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.193910 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" Oct 30 22:21:26.194074 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.193956 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" event={"ID":"d4a885f9-5fd5-4782-978b-ceb9fb740610","Type":"ContainerDied","Data":"0c659a0ec05b303725b1b8ba27a92dfc351c1d382bc21a171eb29baf8eca6e2f"} Oct 30 22:21:26.194074 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.194043 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw" event={"ID":"d4a885f9-5fd5-4782-978b-ceb9fb740610","Type":"ContainerDied","Data":"58ecda98cbb10e4487ed84e0b18b47dd9d630436ab8111d73ac6b8214cbb2509"} Oct 30 22:21:26.194203 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.194093 6006 scope.go:122] "RemoveContainer" containerID="0c659a0ec05b303725b1b8ba27a92dfc351c1d382bc21a171eb29baf8eca6e2f" Oct 30 22:21:26.196802 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.195979 6006 generic.go:356] "Generic (PLEG): container finished" podID="bdbec1e1-b705-464b-88bd-6b6ddbfb3513" containerID="94858957c1fa3efe4e033dd3c7eb1043af3f042f9e6a331f60c29b01339c6d9b" exitCode=0 Oct 30 22:21:26.196802 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.196039 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" Oct 30 22:21:26.196802 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.196075 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" event={"ID":"bdbec1e1-b705-464b-88bd-6b6ddbfb3513","Type":"ContainerDied","Data":"94858957c1fa3efe4e033dd3c7eb1043af3f042f9e6a331f60c29b01339c6d9b"} Oct 30 22:21:26.196802 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.196140 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942" event={"ID":"bdbec1e1-b705-464b-88bd-6b6ddbfb3513","Type":"ContainerDied","Data":"a2d137330f1dba90db30846f0fb058d6b3f6a96abd5220e135cee4d309ccc2be"} Oct 30 22:21:26.198222 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.197813 6006 generic.go:356] "Generic (PLEG): container finished" podID="34b11b2f-2eb6-4313-9a00-74be3c58fecd" containerID="926b17f945f88db77ff307ac8264332a147db9020a96ccb8eb7508ca3bafa007" exitCode=0 Oct 30 22:21:26.198222 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.197845 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" event={"ID":"34b11b2f-2eb6-4313-9a00-74be3c58fecd","Type":"ContainerDied","Data":"926b17f945f88db77ff307ac8264332a147db9020a96ccb8eb7508ca3bafa007"} Oct 30 22:21:26.198222 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.197859 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" Oct 30 22:21:26.198222 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.197903 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt" event={"ID":"34b11b2f-2eb6-4313-9a00-74be3c58fecd","Type":"ContainerDied","Data":"51802e15e4e95d9356c74dfe3c45a423ef0b3c07efc7d83bb0b5e1eac23fe131"} Oct 30 22:21:26.201489 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.201462 6006 generic.go:356] "Generic (PLEG): container finished" podID="7cce6f74-3f9a-49cf-a594-211f3bb1f432" containerID="69e726c5872b7976fdc0fef0721a30048e24a2ddef9f040cd58341f3e86d8fb8" exitCode=0 Oct 30 22:21:26.201604 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.201571 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" event={"ID":"7cce6f74-3f9a-49cf-a594-211f3bb1f432","Type":"ContainerDied","Data":"69e726c5872b7976fdc0fef0721a30048e24a2ddef9f040cd58341f3e86d8fb8"} Oct 30 22:21:26.201667 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.201610 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" event={"ID":"7cce6f74-3f9a-49cf-a594-211f3bb1f432","Type":"ContainerDied","Data":"ccd212ab3a9de1535295f753211d5ec3c510648757ce5fb5314d97133fdb7190"} Oct 30 22:21:26.201718 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.201680 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm" Oct 30 22:21:26.257743 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.257715 6006 scope.go:122] "RemoveContainer" containerID="0c659a0ec05b303725b1b8ba27a92dfc351c1d382bc21a171eb29baf8eca6e2f" Oct 30 22:21:26.258084 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:21:26.258062 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c659a0ec05b303725b1b8ba27a92dfc351c1d382bc21a171eb29baf8eca6e2f\": container with ID starting with 0c659a0ec05b303725b1b8ba27a92dfc351c1d382bc21a171eb29baf8eca6e2f not found: ID does not exist" containerID="0c659a0ec05b303725b1b8ba27a92dfc351c1d382bc21a171eb29baf8eca6e2f" Oct 30 22:21:26.258135 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.258096 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c659a0ec05b303725b1b8ba27a92dfc351c1d382bc21a171eb29baf8eca6e2f"} err="failed to get container status \"0c659a0ec05b303725b1b8ba27a92dfc351c1d382bc21a171eb29baf8eca6e2f\": rpc error: code = NotFound desc = could not find container \"0c659a0ec05b303725b1b8ba27a92dfc351c1d382bc21a171eb29baf8eca6e2f\": container with ID starting with 0c659a0ec05b303725b1b8ba27a92dfc351c1d382bc21a171eb29baf8eca6e2f not found: ID does not exist" Oct 30 22:21:26.258135 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.258120 6006 scope.go:122] "RemoveContainer" containerID="94858957c1fa3efe4e033dd3c7eb1043af3f042f9e6a331f60c29b01339c6d9b" Oct 30 22:21:26.280336 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.280309 6006 scope.go:122] "RemoveContainer" containerID="94858957c1fa3efe4e033dd3c7eb1043af3f042f9e6a331f60c29b01339c6d9b" Oct 30 22:21:26.280700 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:21:26.280675 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94858957c1fa3efe4e033dd3c7eb1043af3f042f9e6a331f60c29b01339c6d9b\": container with ID starting with 94858957c1fa3efe4e033dd3c7eb1043af3f042f9e6a331f60c29b01339c6d9b not found: ID does not exist" containerID="94858957c1fa3efe4e033dd3c7eb1043af3f042f9e6a331f60c29b01339c6d9b" Oct 30 22:21:26.280813 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.280707 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94858957c1fa3efe4e033dd3c7eb1043af3f042f9e6a331f60c29b01339c6d9b"} err="failed to get container status \"94858957c1fa3efe4e033dd3c7eb1043af3f042f9e6a331f60c29b01339c6d9b\": rpc error: code = NotFound desc = could not find container \"94858957c1fa3efe4e033dd3c7eb1043af3f042f9e6a331f60c29b01339c6d9b\": container with ID starting with 94858957c1fa3efe4e033dd3c7eb1043af3f042f9e6a331f60c29b01339c6d9b not found: ID does not exist" Oct 30 22:21:26.280813 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.280727 6006 scope.go:122] "RemoveContainer" containerID="926b17f945f88db77ff307ac8264332a147db9020a96ccb8eb7508ca3bafa007" Oct 30 22:21:26.291954 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.291934 6006 scope.go:122] "RemoveContainer" containerID="926b17f945f88db77ff307ac8264332a147db9020a96ccb8eb7508ca3bafa007" Oct 30 22:21:26.292272 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:21:26.292238 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"926b17f945f88db77ff307ac8264332a147db9020a96ccb8eb7508ca3bafa007\": container with ID starting with 926b17f945f88db77ff307ac8264332a147db9020a96ccb8eb7508ca3bafa007 not found: ID does not exist" containerID="926b17f945f88db77ff307ac8264332a147db9020a96ccb8eb7508ca3bafa007" Oct 30 22:21:26.292357 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.292282 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"926b17f945f88db77ff307ac8264332a147db9020a96ccb8eb7508ca3bafa007"} err="failed to get container status \"926b17f945f88db77ff307ac8264332a147db9020a96ccb8eb7508ca3bafa007\": rpc error: code = NotFound desc = could not find container \"926b17f945f88db77ff307ac8264332a147db9020a96ccb8eb7508ca3bafa007\": container with ID starting with 926b17f945f88db77ff307ac8264332a147db9020a96ccb8eb7508ca3bafa007 not found: ID does not exist" Oct 30 22:21:26.292357 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.292304 6006 scope.go:122] "RemoveContainer" containerID="69e726c5872b7976fdc0fef0721a30048e24a2ddef9f040cd58341f3e86d8fb8" Oct 30 22:21:26.304579 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.304554 6006 scope.go:122] "RemoveContainer" containerID="69e726c5872b7976fdc0fef0721a30048e24a2ddef9f040cd58341f3e86d8fb8" Oct 30 22:21:26.304904 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:21:26.304885 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69e726c5872b7976fdc0fef0721a30048e24a2ddef9f040cd58341f3e86d8fb8\": container with ID starting with 69e726c5872b7976fdc0fef0721a30048e24a2ddef9f040cd58341f3e86d8fb8 not found: ID does not exist" containerID="69e726c5872b7976fdc0fef0721a30048e24a2ddef9f040cd58341f3e86d8fb8" Oct 30 22:21:26.304981 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.304911 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69e726c5872b7976fdc0fef0721a30048e24a2ddef9f040cd58341f3e86d8fb8"} err="failed to get container status \"69e726c5872b7976fdc0fef0721a30048e24a2ddef9f040cd58341f3e86d8fb8\": rpc error: code = NotFound desc = could not find container \"69e726c5872b7976fdc0fef0721a30048e24a2ddef9f040cd58341f3e86d8fb8\": container with ID starting with 69e726c5872b7976fdc0fef0721a30048e24a2ddef9f040cd58341f3e86d8fb8 not found: ID does not exist" Oct 30 22:21:26.537389 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.537291 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw"] Oct 30 22:21:26.537577 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.537524 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-xqvnw"] Oct 30 22:21:26.647098 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.647060 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942"] Oct 30 22:21:26.655452 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.655423 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-ml942"] Oct 30 22:21:26.766616 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.766579 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt"] Oct 30 22:21:26.770417 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.770390 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-qlswt"] Oct 30 22:21:26.887203 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.887170 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm"] Oct 30 22:21:26.891202 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:26.891175 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-779f32-openshift-default-75ddfd96d9-q76mm"] Oct 30 22:21:27.629538 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:27.629489 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b11b2f-2eb6-4313-9a00-74be3c58fecd" path="/var/lib/kubelet/pods/34b11b2f-2eb6-4313-9a00-74be3c58fecd/volumes" Oct 30 22:21:27.630334 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:27.630306 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cce6f74-3f9a-49cf-a594-211f3bb1f432" path="/var/lib/kubelet/pods/7cce6f74-3f9a-49cf-a594-211f3bb1f432/volumes" Oct 30 22:21:27.630999 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:27.630975 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdbec1e1-b705-464b-88bd-6b6ddbfb3513" path="/var/lib/kubelet/pods/bdbec1e1-b705-464b-88bd-6b6ddbfb3513/volumes" Oct 30 22:21:27.633325 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:27.633300 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4a885f9-5fd5-4782-978b-ceb9fb740610" path="/var/lib/kubelet/pods/d4a885f9-5fd5-4782-978b-ceb9fb740610/volumes" Oct 30 22:21:27.913965 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:21:27.913872 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:21:28.492658 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:28.492620 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:21:32.120513 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:32.120392 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:32.166549 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:32.166496 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:32.408476 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:32.408356 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:32.441403 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:32.441352 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 22:21:32.473397 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:32.473346 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 22:21:32.834277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:32.832227 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:32.834277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:32.833586 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:32.995321 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:32.991138 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:33.150341 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:33.123364 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:33.150341 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:33.124140 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:21:33.182282 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:33.176855 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 22:21:33.222277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:33.183628 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:21:33.422339 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:33.420777 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:33.422339 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:33.420881 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:33.422339 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:33.421303 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:21:33.535274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:33.491386 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:33.559462 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:33.537863 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 22:21:33.559462 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:33.540456 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:21:33.559462 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:33.540519 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:21:33.590276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:33.561467 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:33.920918 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:33.836683 6006 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:33.920918 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:33.836779 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 22:21:33.920918 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:33.836991 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/ready\": context deadline exceeded" Oct 30 22:21:33.965433 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:33.965401 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:21:33.965661 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:33.965649 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:21:33.965777 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:33.965768 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:21:33.965910 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:33.965900 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:21:33.966005 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:33.965997 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:21:33.970629 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:33.970602 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:21:34.048116 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:34.048069 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:21:34.049786 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:34.049761 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:21:34.055986 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:34.055957 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:21:34.072944 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:34.061728 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5513f6aa-4fcp9"] Oct 30 22:21:34.072944 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:34.071049 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5513f6aa-4fcp9"] Oct 30 22:21:35.631179 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:35.631140 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5d0d36c-96b0-4e99-b427-8bfe8a0528d2" path="/var/lib/kubelet/pods/a5d0d36c-96b0-4e99-b427-8bfe8a0528d2/volumes" Oct 30 22:21:41.874235 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:41.874197 6006 generic.go:356] "Generic (PLEG): container finished" podID="aa2bdf13-4ade-4f01-8a3c-7463d62efd49" containerID="9e870379876ff473f4bf057ce3a7611f8d90365794d7086c38c4e32d4d5b3aa4" exitCode=0 Oct 30 22:21:41.874688 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:41.874280 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" event={"ID":"aa2bdf13-4ade-4f01-8a3c-7463d62efd49","Type":"ContainerDied","Data":"9e870379876ff473f4bf057ce3a7611f8d90365794d7086c38c4e32d4d5b3aa4"} Oct 30 22:21:41.874802 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:41.874780 6006 scope.go:122] "RemoveContainer" containerID="9e870379876ff473f4bf057ce3a7611f8d90365794d7086c38c4e32d4d5b3aa4" Oct 30 22:21:41.876565 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:41.876532 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:21:41.876665 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:41.876602 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:21:41.876665 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:41.876637 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:21:42.102574 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:42.085931 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:21:42.883768 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:42.883727 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" event={"ID":"aa2bdf13-4ade-4f01-8a3c-7463d62efd49","Type":"ContainerStarted","Data":"73d1fa5fe2df611adf86e78b3c65bfdb7a51fa8a1d40cc97a2be098d88e30093"} Oct 30 22:21:43.030765 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:43.030713 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:21:43.033323 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:43.032984 6006 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 22:21:43.033323 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:43.033081 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" podUID="aa2bdf13-4ade-4f01-8a3c-7463d62efd49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 22:21:44.030917 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:44.030879 6006 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 22:21:44.031359 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:44.030934 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" podUID="aa2bdf13-4ade-4f01-8a3c-7463d62efd49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 22:21:45.031083 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:45.031045 6006 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 22:21:45.031514 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:45.031107 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" podUID="aa2bdf13-4ade-4f01-8a3c-7463d62efd49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 22:21:46.031727 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:46.031687 6006 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 22:21:46.032355 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:46.032320 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" podUID="aa2bdf13-4ade-4f01-8a3c-7463d62efd49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 22:21:47.075274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:47.074553 6006 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 22:21:47.075274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:47.074621 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" podUID="aa2bdf13-4ade-4f01-8a3c-7463d62efd49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 22:21:48.036473 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:48.036437 6006 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 22:21:48.036782 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:48.036754 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" podUID="aa2bdf13-4ade-4f01-8a3c-7463d62efd49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 22:21:49.036203 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:49.036167 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:21:49.036730 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:49.036704 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:21:49.037518 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:49.037493 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:21:57.916882 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:21:57.916833 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:21:58.394171 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:21:58.394131 6006 scope.go:122] "RemoveContainer" containerID="7d4d39b1a235523ce43c63c6b59e5207dabdf45f9608acf4140c68f93c42e9e2" Oct 30 22:22:27.912777 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:22:27.912733 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:22:33.468155 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:22:33.468112 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:22:57.912066 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:22:57.912024 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:22:58.561786 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:22:58.561750 6006 scope.go:122] "RemoveContainer" containerID="2a4d2162df137a3f2718ded7fdc879ba568e2a57599a3c25316c8f0656a9bf64" Oct 30 22:22:58.572558 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:22:58.572535 6006 scope.go:122] "RemoveContainer" containerID="7fd53d6debbf81df22723f873cfadda0f91d4eed91e9f9a99e93caa155b37982" Oct 30 22:22:58.582871 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:22:58.582847 6006 scope.go:122] "RemoveContainer" containerID="79fbbc6b7c6adbe4084c0abaf349de6faccdc219f76bf75596048712f3a844d9" Oct 30 22:22:58.592774 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:22:58.592749 6006 scope.go:122] "RemoveContainer" containerID="b5fdafc4a4243c7aab7c119da1922d338d25621dbb097b20eda97bbf92c02bc5" Oct 30 22:23:18.432287 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.432180 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4da07739-gvn9v"] Oct 30 22:23:18.432938 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.432917 6006 cpu_manager.go:414] "containerMap: removing container" podUID="34b11b2f-2eb6-4313-9a00-74be3c58fecd" containerName="istio-proxy" Oct 30 22:23:18.433007 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.432938 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b11b2f-2eb6-4313-9a00-74be3c58fecd" containerName="istio-proxy" Oct 30 22:23:18.433007 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.432966 6006 cpu_manager.go:414] "containerMap: removing container" podUID="bdbec1e1-b705-464b-88bd-6b6ddbfb3513" containerName="istio-proxy" Oct 30 22:23:18.433007 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.432972 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdbec1e1-b705-464b-88bd-6b6ddbfb3513" containerName="istio-proxy" Oct 30 22:23:18.433007 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.432993 6006 cpu_manager.go:414] "containerMap: removing container" podUID="7cce6f74-3f9a-49cf-a594-211f3bb1f432" containerName="istio-proxy" Oct 30 22:23:18.433007 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.432998 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cce6f74-3f9a-49cf-a594-211f3bb1f432" containerName="istio-proxy" Oct 30 22:23:18.433007 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.433004 6006 cpu_manager.go:414] "containerMap: removing container" podUID="d4a885f9-5fd5-4782-978b-ceb9fb740610" containerName="istio-proxy" Oct 30 22:23:18.433007 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.433009 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4a885f9-5fd5-4782-978b-ceb9fb740610" containerName="istio-proxy" Oct 30 22:23:18.433412 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.433020 6006 cpu_manager.go:414] "containerMap: removing container" podUID="181c0684-588d-44ed-b0de-8122c234cc64" containerName="revoke-keys" Oct 30 22:23:18.433412 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.433026 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181c0684-588d-44ed-b0de-8122c234cc64" containerName="revoke-keys" Oct 30 22:23:18.433412 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.433127 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdbec1e1-b705-464b-88bd-6b6ddbfb3513" containerName="istio-proxy" Oct 30 22:23:18.433412 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.433139 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cce6f74-3f9a-49cf-a594-211f3bb1f432" containerName="istio-proxy" Oct 30 22:23:18.433412 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.433146 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4a885f9-5fd5-4782-978b-ceb9fb740610" containerName="istio-proxy" Oct 30 22:23:18.433412 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.433152 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="181c0684-588d-44ed-b0de-8122c234cc64" containerName="revoke-keys" Oct 30 22:23:18.433412 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.433161 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b11b2f-2eb6-4313-9a00-74be3c58fecd" containerName="istio-proxy" Oct 30 22:23:18.436752 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.436727 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4da07739-gvn9v" Oct 30 22:23:18.491942 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.491906 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4da07739-gvn9v"] Oct 30 22:23:18.498621 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.498585 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3289c66-2b2c-4a29-9f0b-442f8c01b296-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4da07739-gvn9v\" (UID: \"a3289c66-2b2c-4a29-9f0b-442f8c01b296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4da07739-gvn9v" Oct 30 22:23:18.498835 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.498808 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3289c66-2b2c-4a29-9f0b-442f8c01b296-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4da07739-gvn9v\" (UID: \"a3289c66-2b2c-4a29-9f0b-442f8c01b296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4da07739-gvn9v" Oct 30 22:23:18.498939 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.498919 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxs46\" (UniqueName: \"kubernetes.io/projected/a3289c66-2b2c-4a29-9f0b-442f8c01b296-kube-api-access-nxs46\") pod \"maas-api-revoke-keys-e2e-mt-4da07739-gvn9v\" (UID: \"a3289c66-2b2c-4a29-9f0b-442f8c01b296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4da07739-gvn9v" Oct 30 22:23:18.538097 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.538054 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:23:18.538361 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.538127 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:23:18.600360 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.600324 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3289c66-2b2c-4a29-9f0b-442f8c01b296-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4da07739-gvn9v\" (UID: \"a3289c66-2b2c-4a29-9f0b-442f8c01b296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4da07739-gvn9v" Oct 30 22:23:18.600563 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.600470 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3289c66-2b2c-4a29-9f0b-442f8c01b296-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4da07739-gvn9v\" (UID: \"a3289c66-2b2c-4a29-9f0b-442f8c01b296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4da07739-gvn9v" Oct 30 22:23:18.600563 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.600558 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxs46\" (UniqueName: \"kubernetes.io/projected/a3289c66-2b2c-4a29-9f0b-442f8c01b296-kube-api-access-nxs46\") pod \"maas-api-revoke-keys-e2e-mt-4da07739-gvn9v\" (UID: \"a3289c66-2b2c-4a29-9f0b-442f8c01b296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4da07739-gvn9v" Oct 30 22:23:18.601198 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.601173 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3289c66-2b2c-4a29-9f0b-442f8c01b296-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4da07739-gvn9v\" (UID: \"a3289c66-2b2c-4a29-9f0b-442f8c01b296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4da07739-gvn9v" Oct 30 22:23:18.603205 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.603179 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3289c66-2b2c-4a29-9f0b-442f8c01b296-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4da07739-gvn9v\" (UID: \"a3289c66-2b2c-4a29-9f0b-442f8c01b296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4da07739-gvn9v" Oct 30 22:23:18.663072 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.663039 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxs46\" (UniqueName: \"kubernetes.io/projected/a3289c66-2b2c-4a29-9f0b-442f8c01b296-kube-api-access-nxs46\") pod \"maas-api-revoke-keys-e2e-mt-4da07739-gvn9v\" (UID: \"a3289c66-2b2c-4a29-9f0b-442f8c01b296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4da07739-gvn9v" Oct 30 22:23:18.751671 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:18.751589 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4da07739-gvn9v" Oct 30 22:23:19.142508 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:23:19.142463 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3289c66_2b2c_4a29_9f0b_442f8c01b296.slice/crio-e2a16a6fbadec5c077952225101ee5e1406a9609463aece3b8a94a074df50ee9 WatchSource:0}: Error finding container e2a16a6fbadec5c077952225101ee5e1406a9609463aece3b8a94a074df50ee9: Status 404 returned error can't find the container with id e2a16a6fbadec5c077952225101ee5e1406a9609463aece3b8a94a074df50ee9 Oct 30 22:23:19.146076 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:19.146051 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4da07739-gvn9v"] Oct 30 22:23:19.950207 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:19.950166 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4da07739-gvn9v" event={"ID":"a3289c66-2b2c-4a29-9f0b-442f8c01b296","Type":"ContainerStarted","Data":"d0e1a0dce780d642d3e38175c21f2a3a8cd2ddc9359859cee77f89b4d82d4bed"} Oct 30 22:23:19.950759 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:19.950230 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4da07739-gvn9v" event={"ID":"a3289c66-2b2c-4a29-9f0b-442f8c01b296","Type":"ContainerStarted","Data":"e2a16a6fbadec5c077952225101ee5e1406a9609463aece3b8a94a074df50ee9"} Oct 30 22:23:20.077167 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:20.077008 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4da07739-gvn9v" podStartSLOduration=2.076990868 podStartE2EDuration="2.076990868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:20.072384313 +0000 UTC m=+2002.624560614" watchObservedRunningTime="2026-10-30 22:23:20.076990868 +0000 UTC m=+2002.629167169" Oct 30 22:23:20.964648 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:20.964606 6006 generic.go:356] "Generic (PLEG): container finished" podID="a3289c66-2b2c-4a29-9f0b-442f8c01b296" containerID="d0e1a0dce780d642d3e38175c21f2a3a8cd2ddc9359859cee77f89b4d82d4bed" exitCode=0 Oct 30 22:23:20.965169 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:20.964666 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4da07739-gvn9v" event={"ID":"a3289c66-2b2c-4a29-9f0b-442f8c01b296","Type":"ContainerDied","Data":"d0e1a0dce780d642d3e38175c21f2a3a8cd2ddc9359859cee77f89b4d82d4bed"} Oct 30 22:23:21.260976 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:21.260892 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj"] Oct 30 22:23:21.261135 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:21.261090 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" podUID="7acaef50-e202-45c0-ad2a-182925d51e0f" containerName="istio-proxy" containerID="cri-o://dc3ecd21dbcae16625544626920d900e86139e2f5c9f7a5da82e190c248ee51f" gracePeriod=30 Oct 30 22:23:22.157562 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:22.157535 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4da07739-gvn9v" Oct 30 22:23:22.288983 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:22.288884 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3289c66-2b2c-4a29-9f0b-442f8c01b296-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3289c66-2b2c-4a29-9f0b-442f8c01b296-maas-api-xks-ca\") pod \"a3289c66-2b2c-4a29-9f0b-442f8c01b296\" (UID: \"a3289c66-2b2c-4a29-9f0b-442f8c01b296\") " Oct 30 22:23:22.289157 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:22.288991 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3289c66-2b2c-4a29-9f0b-442f8c01b296-kube-api-access-nxs46\" (UniqueName: \"kubernetes.io/projected/a3289c66-2b2c-4a29-9f0b-442f8c01b296-kube-api-access-nxs46\") pod \"a3289c66-2b2c-4a29-9f0b-442f8c01b296\" (UID: \"a3289c66-2b2c-4a29-9f0b-442f8c01b296\") " Oct 30 22:23:22.289157 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:22.289031 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3289c66-2b2c-4a29-9f0b-442f8c01b296-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3289c66-2b2c-4a29-9f0b-442f8c01b296-maas-api-openshift-ca\") pod \"a3289c66-2b2c-4a29-9f0b-442f8c01b296\" (UID: \"a3289c66-2b2c-4a29-9f0b-442f8c01b296\") " Oct 30 22:23:22.290148 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:22.290115 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3289c66-2b2c-4a29-9f0b-442f8c01b296-maas-api-openshift-ca" pod "a3289c66-2b2c-4a29-9f0b-442f8c01b296" (UID: "a3289c66-2b2c-4a29-9f0b-442f8c01b296"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:22.291696 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:22.291652 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3289c66-2b2c-4a29-9f0b-442f8c01b296-maas-api-xks-ca" pod "a3289c66-2b2c-4a29-9f0b-442f8c01b296" (UID: "a3289c66-2b2c-4a29-9f0b-442f8c01b296"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:23:22.292709 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:22.292677 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3289c66-2b2c-4a29-9f0b-442f8c01b296-kube-api-access-nxs46" pod "a3289c66-2b2c-4a29-9f0b-442f8c01b296" (UID: "a3289c66-2b2c-4a29-9f0b-442f8c01b296"). InnerVolumeSpecName "kube-api-access-nxs46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:22.391059 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:22.391018 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3289c66-2b2c-4a29-9f0b-442f8c01b296-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:22.391059 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:22.391052 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxs46\" (UniqueName: \"kubernetes.io/projected/a3289c66-2b2c-4a29-9f0b-442f8c01b296-kube-api-access-nxs46\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:22.391059 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:22.391064 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3289c66-2b2c-4a29-9f0b-442f8c01b296-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:22.984435 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:22.984399 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4da07739-gvn9v" Oct 30 22:23:22.984435 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:22.984420 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4da07739-gvn9v" event={"ID":"a3289c66-2b2c-4a29-9f0b-442f8c01b296","Type":"ContainerDied","Data":"e2a16a6fbadec5c077952225101ee5e1406a9609463aece3b8a94a074df50ee9"} Oct 30 22:23:22.984654 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:22.984483 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2a16a6fbadec5c077952225101ee5e1406a9609463aece3b8a94a074df50ee9" Oct 30 22:23:23.039784 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:23.039750 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf"] Oct 30 22:23:23.039970 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:23.039950 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" podUID="8033a54b-e69c-49d8-ad42-315c1486179a" containerName="istio-proxy" containerID="cri-o://8fe714b8954764a1a65c00861a029ef689ea1b8e7472eb57fa355401242fe705" gracePeriod=30 Oct 30 22:23:25.427350 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:25.427313 6006 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:23:25.427849 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:25.427406 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" podUID="8033a54b-e69c-49d8-ad42-315c1486179a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:23:26.290014 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:26.289974 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 22:23:26.290292 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:26.290059 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" podUID="7acaef50-e202-45c0-ad2a-182925d51e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 22:23:26.882272 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:26.876450 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:23:27.021295 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.021156 6006 generic.go:356] "Generic (PLEG): container finished" podID="7acaef50-e202-45c0-ad2a-182925d51e0f" containerID="dc3ecd21dbcae16625544626920d900e86139e2f5c9f7a5da82e190c248ee51f" exitCode=0 Oct 30 22:23:27.021295 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.021226 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" Oct 30 22:23:27.021295 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.021239 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" event={"ID":"7acaef50-e202-45c0-ad2a-182925d51e0f","Type":"ContainerDied","Data":"dc3ecd21dbcae16625544626920d900e86139e2f5c9f7a5da82e190c248ee51f"} Oct 30 22:23:27.021590 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.021340 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj" event={"ID":"7acaef50-e202-45c0-ad2a-182925d51e0f","Type":"ContainerDied","Data":"c17705a08f0c8c3ce99b1fcc62e26ddcaac293dbd6d0cca7eceadf4eab090f5d"} Oct 30 22:23:27.021590 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.021369 6006 scope.go:122] "RemoveContainer" containerID="dc3ecd21dbcae16625544626920d900e86139e2f5c9f7a5da82e190c248ee51f" Oct 30 22:23:27.035812 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.035786 6006 scope.go:122] "RemoveContainer" containerID="dc3ecd21dbcae16625544626920d900e86139e2f5c9f7a5da82e190c248ee51f" Oct 30 22:23:27.036188 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:23:27.036161 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc3ecd21dbcae16625544626920d900e86139e2f5c9f7a5da82e190c248ee51f\": container with ID starting with dc3ecd21dbcae16625544626920d900e86139e2f5c9f7a5da82e190c248ee51f not found: ID does not exist" containerID="dc3ecd21dbcae16625544626920d900e86139e2f5c9f7a5da82e190c248ee51f" Oct 30 22:23:27.036379 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.036194 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc3ecd21dbcae16625544626920d900e86139e2f5c9f7a5da82e190c248ee51f"} err="failed to get container status \"dc3ecd21dbcae16625544626920d900e86139e2f5c9f7a5da82e190c248ee51f\": rpc error: code = NotFound desc = could not find container \"dc3ecd21dbcae16625544626920d900e86139e2f5c9f7a5da82e190c248ee51f\": container with ID starting with dc3ecd21dbcae16625544626920d900e86139e2f5c9f7a5da82e190c248ee51f not found: ID does not exist" Oct 30 22:23:27.043029 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.042996 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7acaef50-e202-45c0-ad2a-182925d51e0f-kube-api-access-5smz4\" (UniqueName: \"kubernetes.io/projected/7acaef50-e202-45c0-ad2a-182925d51e0f-kube-api-access-5smz4\") pod \"7acaef50-e202-45c0-ad2a-182925d51e0f\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " Oct 30 22:23:27.043180 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.043102 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-envoy\") pod \"7acaef50-e202-45c0-ad2a-182925d51e0f\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " Oct 30 22:23:27.043180 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.043160 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-workload-socket\") pod \"7acaef50-e202-45c0-ad2a-182925d51e0f\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " Oct 30 22:23:27.043295 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.043211 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-token\" (UniqueName: \"kubernetes.io/projected/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-token\") pod \"7acaef50-e202-45c0-ad2a-182925d51e0f\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " Oct 30 22:23:27.043295 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.043285 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-data\") pod \"7acaef50-e202-45c0-ad2a-182925d51e0f\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " Oct 30 22:23:27.043455 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.043438 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7acaef50-e202-45c0-ad2a-182925d51e0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7acaef50-e202-45c0-ad2a-182925d51e0f-istiod-ca-cert\") pod \"7acaef50-e202-45c0-ad2a-182925d51e0f\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " Oct 30 22:23:27.043592 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.043576 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-credential-socket\") pod \"7acaef50-e202-45c0-ad2a-182925d51e0f\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " Oct 30 22:23:27.043652 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.043638 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-podinfo\") pod \"7acaef50-e202-45c0-ad2a-182925d51e0f\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " Oct 30 22:23:27.043712 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.043670 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-workload-certs\") pod \"7acaef50-e202-45c0-ad2a-182925d51e0f\" (UID: \"7acaef50-e202-45c0-ad2a-182925d51e0f\") " Oct 30 22:23:27.044549 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.043992 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-data" pod "7acaef50-e202-45c0-ad2a-182925d51e0f" (UID: "7acaef50-e202-45c0-ad2a-182925d51e0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:27.044549 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.044060 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-credential-socket" pod "7acaef50-e202-45c0-ad2a-182925d51e0f" (UID: "7acaef50-e202-45c0-ad2a-182925d51e0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:27.044549 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.044074 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-workload-socket" pod "7acaef50-e202-45c0-ad2a-182925d51e0f" (UID: "7acaef50-e202-45c0-ad2a-182925d51e0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:27.044549 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.044363 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-workload-certs" pod "7acaef50-e202-45c0-ad2a-182925d51e0f" (UID: "7acaef50-e202-45c0-ad2a-182925d51e0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:27.044549 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.044396 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7acaef50-e202-45c0-ad2a-182925d51e0f-istiod-ca-cert" pod "7acaef50-e202-45c0-ad2a-182925d51e0f" (UID: "7acaef50-e202-45c0-ad2a-182925d51e0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:27.045061 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.045035 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:27.045189 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.045066 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:27.045189 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.045082 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:27.046102 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.046078 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-envoy" pod "7acaef50-e202-45c0-ad2a-182925d51e0f" (UID: "7acaef50-e202-45c0-ad2a-182925d51e0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:27.046566 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.046542 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-podinfo" pod "7acaef50-e202-45c0-ad2a-182925d51e0f" (UID: "7acaef50-e202-45c0-ad2a-182925d51e0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:23:27.046717 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.046692 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7acaef50-e202-45c0-ad2a-182925d51e0f-kube-api-access-5smz4" pod "7acaef50-e202-45c0-ad2a-182925d51e0f" (UID: "7acaef50-e202-45c0-ad2a-182925d51e0f"). InnerVolumeSpecName "kube-api-access-5smz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:27.048064 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.048039 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-token" pod "7acaef50-e202-45c0-ad2a-182925d51e0f" (UID: "7acaef50-e202-45c0-ad2a-182925d51e0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:27.147244 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.147194 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:27.147244 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.147260 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:27.147244 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.147279 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5smz4\" (UniqueName: \"kubernetes.io/projected/7acaef50-e202-45c0-ad2a-182925d51e0f-kube-api-access-5smz4\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:27.147536 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.147294 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:27.147536 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.147308 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7acaef50-e202-45c0-ad2a-182925d51e0f-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:27.147536 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.147322 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7acaef50-e202-45c0-ad2a-182925d51e0f-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:27.454367 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.454321 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj"] Oct 30 22:23:27.464628 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.464589 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-srzmj"] Oct 30 22:23:27.605102 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:27.605019 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7acaef50-e202-45c0-ad2a-182925d51e0f" path="/var/lib/kubelet/pods/7acaef50-e202-45c0-ad2a-182925d51e0f/volumes" Oct 30 22:23:27.913868 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:23:27.913775 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:23:28.383944 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.383916 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:23:28.467355 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.467311 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8033a54b-e69c-49d8-ad42-315c1486179a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8033a54b-e69c-49d8-ad42-315c1486179a-istio-podinfo\") pod \"8033a54b-e69c-49d8-ad42-315c1486179a\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " Oct 30 22:23:28.467549 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.467391 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8033a54b-e69c-49d8-ad42-315c1486179a-kube-api-access-wfbgn\" (UniqueName: \"kubernetes.io/projected/8033a54b-e69c-49d8-ad42-315c1486179a-kube-api-access-wfbgn\") pod \"8033a54b-e69c-49d8-ad42-315c1486179a\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " Oct 30 22:23:28.467549 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.467453 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-istio-envoy\") pod \"8033a54b-e69c-49d8-ad42-315c1486179a\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " Oct 30 22:23:28.467683 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.467489 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8033a54b-e69c-49d8-ad42-315c1486179a-istio-token\" (UniqueName: \"kubernetes.io/projected/8033a54b-e69c-49d8-ad42-315c1486179a-istio-token\") pod \"8033a54b-e69c-49d8-ad42-315c1486179a\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " Oct 30 22:23:28.467866 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.467585 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-workload-certs\") pod \"8033a54b-e69c-49d8-ad42-315c1486179a\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " Oct 30 22:23:28.468243 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.468188 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-workload-certs" pod "8033a54b-e69c-49d8-ad42-315c1486179a" (UID: "8033a54b-e69c-49d8-ad42-315c1486179a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:28.468868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.468544 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-istio-data\") pod \"8033a54b-e69c-49d8-ad42-315c1486179a\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " Oct 30 22:23:28.468868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.468827 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-istio-data" pod "8033a54b-e69c-49d8-ad42-315c1486179a" (UID: "8033a54b-e69c-49d8-ad42-315c1486179a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:28.471601 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.471444 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8033a54b-e69c-49d8-ad42-315c1486179a-kube-api-access-wfbgn" pod "8033a54b-e69c-49d8-ad42-315c1486179a" (UID: "8033a54b-e69c-49d8-ad42-315c1486179a"). InnerVolumeSpecName "kube-api-access-wfbgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:28.471738 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.471711 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-workload-socket\") pod \"8033a54b-e69c-49d8-ad42-315c1486179a\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " Oct 30 22:23:28.471888 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.471871 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8033a54b-e69c-49d8-ad42-315c1486179a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8033a54b-e69c-49d8-ad42-315c1486179a-istiod-ca-cert\") pod \"8033a54b-e69c-49d8-ad42-315c1486179a\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " Oct 30 22:23:28.471962 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.471928 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-credential-socket\") pod \"8033a54b-e69c-49d8-ad42-315c1486179a\" (UID: \"8033a54b-e69c-49d8-ad42-315c1486179a\") " Oct 30 22:23:28.472811 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.472085 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-workload-socket" pod "8033a54b-e69c-49d8-ad42-315c1486179a" (UID: "8033a54b-e69c-49d8-ad42-315c1486179a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:28.472811 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.472473 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-credential-socket" pod "8033a54b-e69c-49d8-ad42-315c1486179a" (UID: "8033a54b-e69c-49d8-ad42-315c1486179a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:28.472811 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.472683 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8033a54b-e69c-49d8-ad42-315c1486179a-istio-podinfo" pod "8033a54b-e69c-49d8-ad42-315c1486179a" (UID: "8033a54b-e69c-49d8-ad42-315c1486179a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:23:28.473026 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.472846 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:28.473026 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.472864 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:28.473026 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.472881 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:28.473026 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.472896 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8033a54b-e69c-49d8-ad42-315c1486179a-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:28.473026 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.472910 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfbgn\" (UniqueName: \"kubernetes.io/projected/8033a54b-e69c-49d8-ad42-315c1486179a-kube-api-access-wfbgn\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:28.473026 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.472924 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:28.473026 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.473024 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8033a54b-e69c-49d8-ad42-315c1486179a-istio-token" pod "8033a54b-e69c-49d8-ad42-315c1486179a" (UID: "8033a54b-e69c-49d8-ad42-315c1486179a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:28.475630 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.473165 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8033a54b-e69c-49d8-ad42-315c1486179a-istiod-ca-cert" pod "8033a54b-e69c-49d8-ad42-315c1486179a" (UID: "8033a54b-e69c-49d8-ad42-315c1486179a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:28.475630 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.473625 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-istio-envoy" pod "8033a54b-e69c-49d8-ad42-315c1486179a" (UID: "8033a54b-e69c-49d8-ad42-315c1486179a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:28.574569 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.574533 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8033a54b-e69c-49d8-ad42-315c1486179a-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:28.574569 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.574564 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8033a54b-e69c-49d8-ad42-315c1486179a-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:28.574569 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.574574 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8033a54b-e69c-49d8-ad42-315c1486179a-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:28.806981 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:28.806935 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk"] Oct 30 22:23:29.065724 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:29.065626 6006 generic.go:356] "Generic (PLEG): container finished" podID="8033a54b-e69c-49d8-ad42-315c1486179a" containerID="8fe714b8954764a1a65c00861a029ef689ea1b8e7472eb57fa355401242fe705" exitCode=0 Oct 30 22:23:29.065724 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:29.065704 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" Oct 30 22:23:29.066276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:29.065743 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" event={"ID":"8033a54b-e69c-49d8-ad42-315c1486179a","Type":"ContainerDied","Data":"8fe714b8954764a1a65c00861a029ef689ea1b8e7472eb57fa355401242fe705"} Oct 30 22:23:29.066276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:29.065811 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf" event={"ID":"8033a54b-e69c-49d8-ad42-315c1486179a","Type":"ContainerDied","Data":"d721cc7cfd230de0adbba8bfea2c55a724815412002d2cafb83109af366873ee"} Oct 30 22:23:29.066276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:29.065841 6006 scope.go:122] "RemoveContainer" containerID="8fe714b8954764a1a65c00861a029ef689ea1b8e7472eb57fa355401242fe705" Oct 30 22:23:29.083752 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:29.083647 6006 scope.go:122] "RemoveContainer" containerID="8fe714b8954764a1a65c00861a029ef689ea1b8e7472eb57fa355401242fe705" Oct 30 22:23:29.084132 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:23:29.084094 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fe714b8954764a1a65c00861a029ef689ea1b8e7472eb57fa355401242fe705\": container with ID starting with 8fe714b8954764a1a65c00861a029ef689ea1b8e7472eb57fa355401242fe705 not found: ID does not exist" containerID="8fe714b8954764a1a65c00861a029ef689ea1b8e7472eb57fa355401242fe705" Oct 30 22:23:29.084241 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:29.084143 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe714b8954764a1a65c00861a029ef689ea1b8e7472eb57fa355401242fe705"} err="failed to get container status \"8fe714b8954764a1a65c00861a029ef689ea1b8e7472eb57fa355401242fe705\": rpc error: code = NotFound desc = could not find container \"8fe714b8954764a1a65c00861a029ef689ea1b8e7472eb57fa355401242fe705\": container with ID starting with 8fe714b8954764a1a65c00861a029ef689ea1b8e7472eb57fa355401242fe705 not found: ID does not exist" Oct 30 22:23:29.238747 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:29.238707 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf"] Oct 30 22:23:29.245459 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:29.245406 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-x9lkf"] Oct 30 22:23:29.606718 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:29.606677 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8033a54b-e69c-49d8-ad42-315c1486179a" path="/var/lib/kubelet/pods/8033a54b-e69c-49d8-ad42-315c1486179a/volumes" Oct 30 22:23:33.807368 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:33.807316 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk" podUID="7ddedf10-57e9-42d9-b1e7-3be877aba9d4" containerName="payload-processing" containerID="cri-o://29dbcd9bdf8413a1e1fa4f053a60542a79aa9a7c1cfe594b0a9c09eefa2e5adb" gracePeriod=25 Oct 30 22:23:34.105965 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:34.105931 6006 generic.go:356] "Generic (PLEG): container finished" podID="7ddedf10-57e9-42d9-b1e7-3be877aba9d4" containerID="29dbcd9bdf8413a1e1fa4f053a60542a79aa9a7c1cfe594b0a9c09eefa2e5adb" exitCode=0 Oct 30 22:23:34.106136 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:34.106015 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk" event={"ID":"7ddedf10-57e9-42d9-b1e7-3be877aba9d4","Type":"ContainerDied","Data":"29dbcd9bdf8413a1e1fa4f053a60542a79aa9a7c1cfe594b0a9c09eefa2e5adb"} Oct 30 22:23:34.106136 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:34.106083 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk" event={"ID":"7ddedf10-57e9-42d9-b1e7-3be877aba9d4","Type":"ContainerDied","Data":"f932559dcdf9f55511e74032619f4defba5b8ef4d0c17b7272a3d1fb1080f810"} Oct 30 22:23:34.106136 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:34.106106 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f932559dcdf9f55511e74032619f4defba5b8ef4d0c17b7272a3d1fb1080f810" Oct 30 22:23:34.111292 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:34.111260 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk" Oct 30 22:23:34.204468 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:34.204430 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ddedf10-57e9-42d9-b1e7-3be877aba9d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ddedf10-57e9-42d9-b1e7-3be877aba9d4-plugins-config-volume\") pod \"7ddedf10-57e9-42d9-b1e7-3be877aba9d4\" (UID: \"7ddedf10-57e9-42d9-b1e7-3be877aba9d4\") " Oct 30 22:23:34.204659 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:34.204516 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ddedf10-57e9-42d9-b1e7-3be877aba9d4-kube-api-access-6srx5\" (UniqueName: \"kubernetes.io/projected/7ddedf10-57e9-42d9-b1e7-3be877aba9d4-kube-api-access-6srx5\") pod \"7ddedf10-57e9-42d9-b1e7-3be877aba9d4\" (UID: \"7ddedf10-57e9-42d9-b1e7-3be877aba9d4\") " Oct 30 22:23:34.204844 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:34.204821 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ddedf10-57e9-42d9-b1e7-3be877aba9d4-plugins-config-volume" pod "7ddedf10-57e9-42d9-b1e7-3be877aba9d4" (UID: "7ddedf10-57e9-42d9-b1e7-3be877aba9d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:34.205232 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:34.205202 6006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ddedf10-57e9-42d9-b1e7-3be877aba9d4-plugins-config-volume\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:34.206709 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:34.206689 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ddedf10-57e9-42d9-b1e7-3be877aba9d4-kube-api-access-6srx5" pod "7ddedf10-57e9-42d9-b1e7-3be877aba9d4" (UID: "7ddedf10-57e9-42d9-b1e7-3be877aba9d4"). InnerVolumeSpecName "kube-api-access-6srx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:34.307400 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:34.307357 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6srx5\" (UniqueName: \"kubernetes.io/projected/7ddedf10-57e9-42d9-b1e7-3be877aba9d4-kube-api-access-6srx5\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:35.107429 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:35.107330 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk" podUID="7ddedf10-57e9-42d9-b1e7-3be877aba9d4" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.125:50052\" within 1s: context deadline exceeded" Oct 30 22:23:35.107877 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:23:35.107435 6006 logging.go:55] [core] [Channel #1964 SubChannel #1965]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.125:50052", ServerName: "10.132.0.125:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.125:50052: operation was canceled" Oct 30 22:23:35.116284 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:35.116236 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk" Oct 30 22:23:35.300543 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:35.300504 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk"] Oct 30 22:23:35.302524 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:35.302493 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4da07739-84f8c975cc-spsjk"] Oct 30 22:23:35.608414 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:35.608378 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ddedf10-57e9-42d9-b1e7-3be877aba9d4" path="/var/lib/kubelet/pods/7ddedf10-57e9-42d9-b1e7-3be877aba9d4/volumes" Oct 30 22:23:41.097903 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.097871 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f93eab74-nrqt7"] Oct 30 22:23:41.098689 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.098665 6006 cpu_manager.go:414] "containerMap: removing container" podUID="7acaef50-e202-45c0-ad2a-182925d51e0f" containerName="istio-proxy" Oct 30 22:23:41.098739 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.098693 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7acaef50-e202-45c0-ad2a-182925d51e0f" containerName="istio-proxy" Oct 30 22:23:41.098739 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.098735 6006 cpu_manager.go:414] "containerMap: removing container" podUID="a3289c66-2b2c-4a29-9f0b-442f8c01b296" containerName="revoke-keys" Oct 30 22:23:41.098808 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.098742 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3289c66-2b2c-4a29-9f0b-442f8c01b296" containerName="revoke-keys" Oct 30 22:23:41.098838 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.098811 6006 cpu_manager.go:414] "containerMap: removing container" podUID="7ddedf10-57e9-42d9-b1e7-3be877aba9d4" containerName="payload-processing" Oct 30 22:23:41.098838 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.098820 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ddedf10-57e9-42d9-b1e7-3be877aba9d4" containerName="payload-processing" Oct 30 22:23:41.098838 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.098831 6006 cpu_manager.go:414] "containerMap: removing container" podUID="8033a54b-e69c-49d8-ad42-315c1486179a" containerName="istio-proxy" Oct 30 22:23:41.098928 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.098839 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8033a54b-e69c-49d8-ad42-315c1486179a" containerName="istio-proxy" Oct 30 22:23:41.099024 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.099012 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ddedf10-57e9-42d9-b1e7-3be877aba9d4" containerName="payload-processing" Oct 30 22:23:41.099108 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.099029 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="8033a54b-e69c-49d8-ad42-315c1486179a" containerName="istio-proxy" Oct 30 22:23:41.099108 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.099048 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="7acaef50-e202-45c0-ad2a-182925d51e0f" containerName="istio-proxy" Oct 30 22:23:41.099108 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.099061 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3289c66-2b2c-4a29-9f0b-442f8c01b296" containerName="revoke-keys" Oct 30 22:23:41.103646 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.103625 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93eab74-nrqt7" Oct 30 22:23:41.162066 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.162024 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f93eab74-nrqt7"] Oct 30 22:23:41.207375 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.207331 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:23:41.207603 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.207426 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:23:41.256635 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.256596 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b33669b-cc84-48a7-a786-d7689ca52e5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f93eab74-nrqt7\" (UID: \"4b33669b-cc84-48a7-a786-d7689ca52e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93eab74-nrqt7" Oct 30 22:23:41.256817 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.256654 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b33669b-cc84-48a7-a786-d7689ca52e5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f93eab74-nrqt7\" (UID: \"4b33669b-cc84-48a7-a786-d7689ca52e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93eab74-nrqt7" Oct 30 22:23:41.256817 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.256773 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v94nl\" (UniqueName: \"kubernetes.io/projected/4b33669b-cc84-48a7-a786-d7689ca52e5c-kube-api-access-v94nl\") pod \"maas-api-revoke-sub-f93eab74-nrqt7\" (UID: \"4b33669b-cc84-48a7-a786-d7689ca52e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93eab74-nrqt7" Oct 30 22:23:41.360885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.359497 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b33669b-cc84-48a7-a786-d7689ca52e5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f93eab74-nrqt7\" (UID: \"4b33669b-cc84-48a7-a786-d7689ca52e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93eab74-nrqt7" Oct 30 22:23:41.361055 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.360776 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b33669b-cc84-48a7-a786-d7689ca52e5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f93eab74-nrqt7\" (UID: \"4b33669b-cc84-48a7-a786-d7689ca52e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93eab74-nrqt7" Oct 30 22:23:41.361055 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.360985 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b33669b-cc84-48a7-a786-d7689ca52e5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f93eab74-nrqt7\" (UID: \"4b33669b-cc84-48a7-a786-d7689ca52e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93eab74-nrqt7" Oct 30 22:23:41.373854 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.361545 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v94nl\" (UniqueName: \"kubernetes.io/projected/4b33669b-cc84-48a7-a786-d7689ca52e5c-kube-api-access-v94nl\") pod \"maas-api-revoke-sub-f93eab74-nrqt7\" (UID: \"4b33669b-cc84-48a7-a786-d7689ca52e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93eab74-nrqt7" Oct 30 22:23:41.378824 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.378747 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b33669b-cc84-48a7-a786-d7689ca52e5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f93eab74-nrqt7\" (UID: \"4b33669b-cc84-48a7-a786-d7689ca52e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93eab74-nrqt7" Oct 30 22:23:41.430136 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.430096 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v94nl\" (UniqueName: \"kubernetes.io/projected/4b33669b-cc84-48a7-a786-d7689ca52e5c-kube-api-access-v94nl\") pod \"maas-api-revoke-sub-f93eab74-nrqt7\" (UID: \"4b33669b-cc84-48a7-a786-d7689ca52e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93eab74-nrqt7" Oct 30 22:23:41.718667 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:41.718558 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93eab74-nrqt7" Oct 30 22:23:42.161319 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:23:42.161280 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b33669b_cc84_48a7_a786_d7689ca52e5c.slice/crio-f403360418f605d80fcfd22b07ffef70f66b98160e7ba762d6fdbb0471d89f10 WatchSource:0}: Error finding container f403360418f605d80fcfd22b07ffef70f66b98160e7ba762d6fdbb0471d89f10: Status 404 returned error can't find the container with id f403360418f605d80fcfd22b07ffef70f66b98160e7ba762d6fdbb0471d89f10 Oct 30 22:23:42.163747 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:42.163719 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f93eab74-nrqt7"] Oct 30 22:23:42.177298 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:42.177266 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93eab74-nrqt7" event={"ID":"4b33669b-cc84-48a7-a786-d7689ca52e5c","Type":"ContainerStarted","Data":"f403360418f605d80fcfd22b07ffef70f66b98160e7ba762d6fdbb0471d89f10"} Oct 30 22:23:43.186112 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:43.186078 6006 generic.go:356] "Generic (PLEG): container finished" podID="4b33669b-cc84-48a7-a786-d7689ca52e5c" containerID="28b1b1317ac5dfc0ea48b6907219eafc9de4103ece670b7325bdd8f39445e4bb" exitCode=0 Oct 30 22:23:43.186584 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:43.186161 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93eab74-nrqt7" event={"ID":"4b33669b-cc84-48a7-a786-d7689ca52e5c","Type":"ContainerDied","Data":"28b1b1317ac5dfc0ea48b6907219eafc9de4103ece670b7325bdd8f39445e4bb"} Oct 30 22:23:44.384889 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:44.384860 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93eab74-nrqt7" Oct 30 22:23:44.524112 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:44.524003 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b33669b-cc84-48a7-a786-d7689ca52e5c-kube-api-access-v94nl\" (UniqueName: \"kubernetes.io/projected/4b33669b-cc84-48a7-a786-d7689ca52e5c-kube-api-access-v94nl\") pod \"4b33669b-cc84-48a7-a786-d7689ca52e5c\" (UID: \"4b33669b-cc84-48a7-a786-d7689ca52e5c\") " Oct 30 22:23:44.524112 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:44.524074 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b33669b-cc84-48a7-a786-d7689ca52e5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b33669b-cc84-48a7-a786-d7689ca52e5c-maas-api-openshift-ca\") pod \"4b33669b-cc84-48a7-a786-d7689ca52e5c\" (UID: \"4b33669b-cc84-48a7-a786-d7689ca52e5c\") " Oct 30 22:23:44.524376 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:44.524301 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b33669b-cc84-48a7-a786-d7689ca52e5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b33669b-cc84-48a7-a786-d7689ca52e5c-maas-api-xks-ca\") pod \"4b33669b-cc84-48a7-a786-d7689ca52e5c\" (UID: \"4b33669b-cc84-48a7-a786-d7689ca52e5c\") " Oct 30 22:23:44.525664 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:44.525634 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b33669b-cc84-48a7-a786-d7689ca52e5c-maas-api-openshift-ca" pod "4b33669b-cc84-48a7-a786-d7689ca52e5c" (UID: "4b33669b-cc84-48a7-a786-d7689ca52e5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:44.527893 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:44.527866 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b33669b-cc84-48a7-a786-d7689ca52e5c-kube-api-access-v94nl" pod "4b33669b-cc84-48a7-a786-d7689ca52e5c" (UID: "4b33669b-cc84-48a7-a786-d7689ca52e5c"). InnerVolumeSpecName "kube-api-access-v94nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:44.530703 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:44.530673 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b33669b-cc84-48a7-a786-d7689ca52e5c-maas-api-xks-ca" pod "4b33669b-cc84-48a7-a786-d7689ca52e5c" (UID: "4b33669b-cc84-48a7-a786-d7689ca52e5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:23:44.626823 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:44.626785 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b33669b-cc84-48a7-a786-d7689ca52e5c-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:44.626823 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:44.626818 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v94nl\" (UniqueName: \"kubernetes.io/projected/4b33669b-cc84-48a7-a786-d7689ca52e5c-kube-api-access-v94nl\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:44.626823 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:44.626829 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b33669b-cc84-48a7-a786-d7689ca52e5c-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:45.203950 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:45.203916 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93eab74-nrqt7" Oct 30 22:23:45.204111 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:45.203952 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f93eab74-nrqt7" event={"ID":"4b33669b-cc84-48a7-a786-d7689ca52e5c","Type":"ContainerDied","Data":"f403360418f605d80fcfd22b07ffef70f66b98160e7ba762d6fdbb0471d89f10"} Oct 30 22:23:45.204111 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:45.204009 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f403360418f605d80fcfd22b07ffef70f66b98160e7ba762d6fdbb0471d89f10" Oct 30 22:23:48.476508 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:48.476468 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:23:57.914638 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:23:57.914592 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:23:58.636704 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:58.636590 6006 scope.go:122] "RemoveContainer" containerID="566d0ded391a908ff82aa3207a5115a6381963deec5820a8b3b14c96334a75e2" Oct 30 22:23:58.655678 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:23:58.655649 6006 scope.go:122] "RemoveContainer" containerID="956f6bb2952fbf2b9213feb70ce88cc7e3e8b02dc667aa4844c51f441b2978b1" Oct 30 22:24:03.400433 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:03.400397 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4da07739-gvn9v"] Oct 30 22:24:03.451027 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:03.450985 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4da07739-gvn9v"] Oct 30 22:24:03.605692 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:03.605654 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3289c66-2b2c-4a29-9f0b-442f8c01b296" path="/var/lib/kubelet/pods/a3289c66-2b2c-4a29-9f0b-442f8c01b296/volumes" Oct 30 22:24:27.917309 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:24:27.917264 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:24:56.276789 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:56.276521 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn"] Oct 30 22:24:56.279102 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:56.279069 6006 cpu_manager.go:414] "containerMap: removing container" podUID="4b33669b-cc84-48a7-a786-d7689ca52e5c" containerName="revoke-keys" Oct 30 22:24:56.279374 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:56.279342 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b33669b-cc84-48a7-a786-d7689ca52e5c" containerName="revoke-keys" Oct 30 22:24:56.279698 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:56.279679 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b33669b-cc84-48a7-a786-d7689ca52e5c" containerName="revoke-keys" Oct 30 22:24:56.285086 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:56.285056 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn" Oct 30 22:24:56.334964 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:56.334922 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn"] Oct 30 22:24:56.347070 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:56.347033 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0583d379-6fbe-403b-bf73-2dca5514c792-metrics-certs\") pod \"maas-api-e2e-api-a-c24d61-54688689c7-qzwgn\" (UID: \"0583d379-6fbe-403b-bf73-2dca5514c792\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn" Oct 30 22:24:56.347314 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:56.347165 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0583d379-6fbe-403b-bf73-2dca5514c792-openshift-service-ca\") pod \"maas-api-e2e-api-a-c24d61-54688689c7-qzwgn\" (UID: \"0583d379-6fbe-403b-bf73-2dca5514c792\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn" Oct 30 22:24:56.347314 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:56.347190 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0583d379-6fbe-403b-bf73-2dca5514c792-maas-api-tls\") pod \"maas-api-e2e-api-a-c24d61-54688689c7-qzwgn\" (UID: \"0583d379-6fbe-403b-bf73-2dca5514c792\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn" Oct 30 22:24:56.347493 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:56.347352 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz8fz\" (UniqueName: \"kubernetes.io/projected/0583d379-6fbe-403b-bf73-2dca5514c792-kube-api-access-fz8fz\") pod \"maas-api-e2e-api-a-c24d61-54688689c7-qzwgn\" (UID: \"0583d379-6fbe-403b-bf73-2dca5514c792\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn" Oct 30 22:24:56.388000 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:56.387575 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c24d61-serving-cert\"" Oct 30 22:24:56.449354 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:56.449308 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0583d379-6fbe-403b-bf73-2dca5514c792-metrics-certs\") pod \"maas-api-e2e-api-a-c24d61-54688689c7-qzwgn\" (UID: \"0583d379-6fbe-403b-bf73-2dca5514c792\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn" Oct 30 22:24:56.449778 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:56.449451 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0583d379-6fbe-403b-bf73-2dca5514c792-openshift-service-ca\") pod \"maas-api-e2e-api-a-c24d61-54688689c7-qzwgn\" (UID: \"0583d379-6fbe-403b-bf73-2dca5514c792\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn" Oct 30 22:24:56.449778 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:56.449483 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0583d379-6fbe-403b-bf73-2dca5514c792-maas-api-tls\") pod \"maas-api-e2e-api-a-c24d61-54688689c7-qzwgn\" (UID: \"0583d379-6fbe-403b-bf73-2dca5514c792\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn" Oct 30 22:24:56.449778 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:56.449584 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz8fz\" (UniqueName: \"kubernetes.io/projected/0583d379-6fbe-403b-bf73-2dca5514c792-kube-api-access-fz8fz\") pod \"maas-api-e2e-api-a-c24d61-54688689c7-qzwgn\" (UID: \"0583d379-6fbe-403b-bf73-2dca5514c792\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn" Oct 30 22:24:56.450116 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:24:56.450093 6006 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-serving-cert: secret "maas-api-e2e-api-a-c24d61-serving-cert" not found Oct 30 22:24:56.450207 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:24:56.450193 6006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0583d379-6fbe-403b-bf73-2dca5514c792-maas-api-tls podName:0583d379-6fbe-403b-bf73-2dca5514c792 nodeName:}" failed. No retries permitted until 2026-10-30 22:24:56.950171118 +0000 UTC m=+2099.502347417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0583d379-6fbe-403b-bf73-2dca5514c792-maas-api-tls") pod "maas-api-e2e-api-a-c24d61-54688689c7-qzwgn" (UID: "0583d379-6fbe-403b-bf73-2dca5514c792") : secret "maas-api-e2e-api-a-c24d61-serving-cert" not found Oct 30 22:24:56.450419 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:56.450397 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0583d379-6fbe-403b-bf73-2dca5514c792-openshift-service-ca\") pod \"maas-api-e2e-api-a-c24d61-54688689c7-qzwgn\" (UID: \"0583d379-6fbe-403b-bf73-2dca5514c792\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn" Oct 30 22:24:56.452182 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:56.452158 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0583d379-6fbe-403b-bf73-2dca5514c792-metrics-certs\") pod \"maas-api-e2e-api-a-c24d61-54688689c7-qzwgn\" (UID: \"0583d379-6fbe-403b-bf73-2dca5514c792\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn" Oct 30 22:24:56.553616 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:56.553581 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz8fz\" (UniqueName: \"kubernetes.io/projected/0583d379-6fbe-403b-bf73-2dca5514c792-kube-api-access-fz8fz\") pod \"maas-api-e2e-api-a-c24d61-54688689c7-qzwgn\" (UID: \"0583d379-6fbe-403b-bf73-2dca5514c792\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn" Oct 30 22:24:56.957186 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:56.957090 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0583d379-6fbe-403b-bf73-2dca5514c792-maas-api-tls\") pod \"maas-api-e2e-api-a-c24d61-54688689c7-qzwgn\" (UID: \"0583d379-6fbe-403b-bf73-2dca5514c792\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn" Oct 30 22:24:56.959949 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:56.959915 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0583d379-6fbe-403b-bf73-2dca5514c792-maas-api-tls\") pod \"maas-api-e2e-api-a-c24d61-54688689c7-qzwgn\" (UID: \"0583d379-6fbe-403b-bf73-2dca5514c792\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn" Oct 30 22:24:57.200713 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:57.200660 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn" Oct 30 22:24:57.588983 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:24:57.588945 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0583d379_6fbe_403b_bf73_2dca5514c792.slice/crio-c3cadeb89f370c94595a531cfec3029009d3a31ef4c958d75910379c2d95f8fa WatchSource:0}: Error finding container c3cadeb89f370c94595a531cfec3029009d3a31ef4c958d75910379c2d95f8fa: Status 404 returned error can't find the container with id c3cadeb89f370c94595a531cfec3029009d3a31ef4c958d75910379c2d95f8fa Oct 30 22:24:57.590429 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:57.590407 6006 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:24:57.590679 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:57.590657 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn"] Oct 30 22:24:57.915112 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:24:57.915023 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:24:58.002037 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:58.002004 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 22:24:58.016139 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:58.016106 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 22:24:58.179388 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:58.179287 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn" event={"ID":"0583d379-6fbe-403b-bf73-2dca5514c792","Type":"ContainerStarted","Data":"c3cadeb89f370c94595a531cfec3029009d3a31ef4c958d75910379c2d95f8fa"} Oct 30 22:24:59.218186 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:59.218129 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn" event={"ID":"0583d379-6fbe-403b-bf73-2dca5514c792","Type":"ContainerStarted","Data":"73abe01db0c06a3a42c0def225e25e7b6e7ce9308091fdf87f772708b0a8a05d"} Oct 30 22:24:59.220168 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:59.220135 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn" Oct 30 22:24:59.393133 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:24:59.393074 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn" podStartSLOduration=2.554275522 podStartE2EDuration="3.393056759s" totalImagesPullingTime="838.781237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:59.392092318 +0000 UTC m=+2101.944268630" watchObservedRunningTime="2026-10-30 22:24:59.393056759 +0000 UTC m=+2101.945233114" Oct 30 22:25:06.253441 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:06.253410 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn" Oct 30 22:25:08.778521 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:08.778484 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn"] Oct 30 22:25:08.778982 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:08.778702 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" podUID="67ba26ec-c30a-46f7-8b3d-d6ef88849246" containerName="istio-proxy" containerID="cri-o://28f04748c2990fc6b8c23ed57f6df2026b60a46591783d8fe22d8c2e156df2f0" gracePeriod=30 Oct 30 22:25:08.831932 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:08.831885 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv"] Oct 30 22:25:08.832898 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:08.832165 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" podUID="4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6" containerName="istio-proxy" containerID="cri-o://1c8c6a047107f5731955e18abb5ab34dee947652edcb3fa9e97321ac5105ed23" gracePeriod=30 Oct 30 22:25:08.905770 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:08.905732 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf"] Oct 30 22:25:08.906062 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:08.906033 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" podUID="492acb10-fd86-4613-a1fc-68a0701b7846" containerName="istio-proxy" containerID="cri-o://50c3fe8a4021575af554f8504c4e041e1c32f8e77051a3555055129cf20d36e3" gracePeriod=30 Oct 30 22:25:08.933066 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:08.933030 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6"] Oct 30 22:25:08.933515 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:08.933479 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" podUID="2cb25e8f-893d-486f-aed7-922d9804d9c9" containerName="istio-proxy" containerID="cri-o://b9e4c73a51ed3bae59bcc8585870682149154c76278a5a0ff30624a82693e525" gracePeriod=30 Oct 30 22:25:08.981043 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:08.981006 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2"] Oct 30 22:25:08.981317 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:08.981291 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" podUID="a852f49d-5ed1-472d-8fb0-f811bbff7e4f" containerName="istio-proxy" containerID="cri-o://1070d264a2421a18dfb291e0a9ba378427eef18cc6a07b8df53c4badd4b28a2e" gracePeriod=30 Oct 30 22:25:08.997012 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:08.996968 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p"] Oct 30 22:25:08.997227 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:08.997203 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" podUID="5aa0ffc2-0a81-47ff-83c5-751eff96ac2d" containerName="istio-proxy" containerID="cri-o://69b2526d6aa09976a97f48bd39c2752b245fc511fe5ddbfb1b8126795ebf3f2f" gracePeriod=30 Oct 30 22:25:09.083010 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:09.082970 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc"] Oct 30 22:25:09.083286 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:09.083229 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" podUID="46a061d3-8105-4260-9bf1-4fc3a5a09343" containerName="istio-proxy" containerID="cri-o://fbc2fbcbea5902da8911c902b7dc076c87011d543a001f4730bd3588bacc4790" gracePeriod=30 Oct 30 22:25:09.126241 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:09.126185 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt"] Oct 30 22:25:09.126591 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:09.126522 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" podUID="bf391b9e-a66a-4b60-9c45-8025b5155829" containerName="istio-proxy" containerID="cri-o://a15439709ea391e6878e0220c6ba1ea463681ea21e3a3372672298292b75475e" gracePeriod=30 Oct 30 22:25:09.153637 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:09.153601 6006 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:25:09.153789 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:09.153661 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" podUID="bf391b9e-a66a-4b60-9c45-8025b5155829" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:25:10.444104 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:10.444051 6006 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:25:10.444636 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:10.444130 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" podUID="5aa0ffc2-0a81-47ff-83c5-751eff96ac2d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:25:10.796992 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:10.796917 6006 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:25:10.796992 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:10.796988 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" podUID="4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:25:11.812983 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:11.812935 6006 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:25:11.813550 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:11.813006 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" podUID="492acb10-fd86-4613-a1fc-68a0701b7846" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:25:12.096543 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:12.096456 6006 patch_prober.go:29] interesting pod/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:25:12.096690 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:12.096533 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" podUID="46a061d3-8105-4260-9bf1-4fc3a5a09343" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:25:12.742961 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:12.742921 6006 patch_prober.go:29] interesting pod/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:25:12.743176 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:12.742979 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" podUID="2cb25e8f-893d-486f-aed7-922d9804d9c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:25:13.618656 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:13.618614 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:25:13.683631 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:13.683590 6006 patch_prober.go:29] interesting pod/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:25:13.683822 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:13.683662 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" podUID="67ba26ec-c30a-46f7-8b3d-d6ef88849246" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:25:13.759864 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:13.759824 6006 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:25:13.760042 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:13.759906 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" podUID="a852f49d-5ed1-472d-8fb0-f811bbff7e4f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:25:14.295291 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.295228 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:25:14.407967 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.406299 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istiod-ca-cert\") pod \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " Oct 30 22:25:14.407967 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.406368 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-podinfo\") pod \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " Oct 30 22:25:14.407967 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.406409 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-envoy\") pod \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " Oct 30 22:25:14.407967 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.406573 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-data\") pod \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " Oct 30 22:25:14.407967 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.406688 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-workload-socket\") pod \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " Oct 30 22:25:14.407967 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.406737 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-workload-certs\") pod \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " Oct 30 22:25:14.407967 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.406799 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-token\" (UniqueName: \"kubernetes.io/projected/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-token\") pod \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " Oct 30 22:25:14.407967 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.406862 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-credential-socket\") pod \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " Oct 30 22:25:14.407967 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.406995 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67ba26ec-c30a-46f7-8b3d-d6ef88849246-kube-api-access-z4h8l\" (UniqueName: \"kubernetes.io/projected/67ba26ec-c30a-46f7-8b3d-d6ef88849246-kube-api-access-z4h8l\") pod \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\" (UID: \"67ba26ec-c30a-46f7-8b3d-d6ef88849246\") " Oct 30 22:25:14.411378 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.410691 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-workload-socket" pod "67ba26ec-c30a-46f7-8b3d-d6ef88849246" (UID: "67ba26ec-c30a-46f7-8b3d-d6ef88849246"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.411378 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.410730 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-data" pod "67ba26ec-c30a-46f7-8b3d-d6ef88849246" (UID: "67ba26ec-c30a-46f7-8b3d-d6ef88849246"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.411378 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.411006 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istiod-ca-cert" pod "67ba26ec-c30a-46f7-8b3d-d6ef88849246" (UID: "67ba26ec-c30a-46f7-8b3d-d6ef88849246"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:14.411378 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.411286 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-workload-certs" pod "67ba26ec-c30a-46f7-8b3d-d6ef88849246" (UID: "67ba26ec-c30a-46f7-8b3d-d6ef88849246"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.411378 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.411345 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-credential-socket" pod "67ba26ec-c30a-46f7-8b3d-d6ef88849246" (UID: "67ba26ec-c30a-46f7-8b3d-d6ef88849246"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.425276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.423624 6006 generic.go:356] "Generic (PLEG): container finished" podID="46a061d3-8105-4260-9bf1-4fc3a5a09343" containerID="fbc2fbcbea5902da8911c902b7dc076c87011d543a001f4730bd3588bacc4790" exitCode=0 Oct 30 22:25:14.425276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.423746 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" event={"ID":"46a061d3-8105-4260-9bf1-4fc3a5a09343","Type":"ContainerDied","Data":"fbc2fbcbea5902da8911c902b7dc076c87011d543a001f4730bd3588bacc4790"} Oct 30 22:25:14.430703 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.430663 6006 generic.go:356] "Generic (PLEG): container finished" podID="bf391b9e-a66a-4b60-9c45-8025b5155829" containerID="a15439709ea391e6878e0220c6ba1ea463681ea21e3a3372672298292b75475e" exitCode=0 Oct 30 22:25:14.430840 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.430758 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" event={"ID":"bf391b9e-a66a-4b60-9c45-8025b5155829","Type":"ContainerDied","Data":"a15439709ea391e6878e0220c6ba1ea463681ea21e3a3372672298292b75475e"} Oct 30 22:25:14.433262 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.432299 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-podinfo" pod "67ba26ec-c30a-46f7-8b3d-d6ef88849246" (UID: "67ba26ec-c30a-46f7-8b3d-d6ef88849246"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:25:14.433262 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.432483 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ba26ec-c30a-46f7-8b3d-d6ef88849246-kube-api-access-z4h8l" pod "67ba26ec-c30a-46f7-8b3d-d6ef88849246" (UID: "67ba26ec-c30a-46f7-8b3d-d6ef88849246"). InnerVolumeSpecName "kube-api-access-z4h8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:14.433262 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.433037 6006 generic.go:356] "Generic (PLEG): container finished" podID="492acb10-fd86-4613-a1fc-68a0701b7846" containerID="50c3fe8a4021575af554f8504c4e041e1c32f8e77051a3555055129cf20d36e3" exitCode=0 Oct 30 22:25:14.433262 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.433141 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" event={"ID":"492acb10-fd86-4613-a1fc-68a0701b7846","Type":"ContainerDied","Data":"50c3fe8a4021575af554f8504c4e041e1c32f8e77051a3555055129cf20d36e3"} Oct 30 22:25:14.436564 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.436161 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-envoy" pod "67ba26ec-c30a-46f7-8b3d-d6ef88849246" (UID: "67ba26ec-c30a-46f7-8b3d-d6ef88849246"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.436564 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.436233 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-token" pod "67ba26ec-c30a-46f7-8b3d-d6ef88849246" (UID: "67ba26ec-c30a-46f7-8b3d-d6ef88849246"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:14.437146 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.437117 6006 generic.go:356] "Generic (PLEG): container finished" podID="67ba26ec-c30a-46f7-8b3d-d6ef88849246" containerID="28f04748c2990fc6b8c23ed57f6df2026b60a46591783d8fe22d8c2e156df2f0" exitCode=0 Oct 30 22:25:14.437765 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.437384 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" event={"ID":"67ba26ec-c30a-46f7-8b3d-d6ef88849246","Type":"ContainerDied","Data":"28f04748c2990fc6b8c23ed57f6df2026b60a46591783d8fe22d8c2e156df2f0"} Oct 30 22:25:14.437765 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.437652 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" event={"ID":"67ba26ec-c30a-46f7-8b3d-d6ef88849246","Type":"ContainerDied","Data":"b9956bc7e2ebc99a0d5965a6faa759071d1c9ef2b6cf8bbc8ed41a61cb1eda83"} Oct 30 22:25:14.437765 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.437406 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn" Oct 30 22:25:14.437765 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.437686 6006 scope.go:122] "RemoveContainer" containerID="28f04748c2990fc6b8c23ed57f6df2026b60a46591783d8fe22d8c2e156df2f0" Oct 30 22:25:14.451090 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.450887 6006 generic.go:356] "Generic (PLEG): container finished" podID="5aa0ffc2-0a81-47ff-83c5-751eff96ac2d" containerID="69b2526d6aa09976a97f48bd39c2752b245fc511fe5ddbfb1b8126795ebf3f2f" exitCode=0 Oct 30 22:25:14.451090 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.451001 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" event={"ID":"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d","Type":"ContainerDied","Data":"69b2526d6aa09976a97f48bd39c2752b245fc511fe5ddbfb1b8126795ebf3f2f"} Oct 30 22:25:14.453491 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.453321 6006 generic.go:356] "Generic (PLEG): container finished" podID="4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6" containerID="1c8c6a047107f5731955e18abb5ab34dee947652edcb3fa9e97321ac5105ed23" exitCode=0 Oct 30 22:25:14.453491 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.453413 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" event={"ID":"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6","Type":"ContainerDied","Data":"1c8c6a047107f5731955e18abb5ab34dee947652edcb3fa9e97321ac5105ed23"} Oct 30 22:25:14.468745 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.467700 6006 generic.go:356] "Generic (PLEG): container finished" podID="2cb25e8f-893d-486f-aed7-922d9804d9c9" containerID="b9e4c73a51ed3bae59bcc8585870682149154c76278a5a0ff30624a82693e525" exitCode=0 Oct 30 22:25:14.468745 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.467746 6006 scope.go:122] "RemoveContainer" containerID="28f04748c2990fc6b8c23ed57f6df2026b60a46591783d8fe22d8c2e156df2f0" Oct 30 22:25:14.468745 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.467812 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" event={"ID":"2cb25e8f-893d-486f-aed7-922d9804d9c9","Type":"ContainerDied","Data":"b9e4c73a51ed3bae59bcc8585870682149154c76278a5a0ff30624a82693e525"} Oct 30 22:25:14.472659 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:25:14.469444 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28f04748c2990fc6b8c23ed57f6df2026b60a46591783d8fe22d8c2e156df2f0\": container with ID starting with 28f04748c2990fc6b8c23ed57f6df2026b60a46591783d8fe22d8c2e156df2f0 not found: ID does not exist" containerID="28f04748c2990fc6b8c23ed57f6df2026b60a46591783d8fe22d8c2e156df2f0" Oct 30 22:25:14.472659 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.469486 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28f04748c2990fc6b8c23ed57f6df2026b60a46591783d8fe22d8c2e156df2f0"} err="failed to get container status \"28f04748c2990fc6b8c23ed57f6df2026b60a46591783d8fe22d8c2e156df2f0\": rpc error: code = NotFound desc = could not find container \"28f04748c2990fc6b8c23ed57f6df2026b60a46591783d8fe22d8c2e156df2f0\": container with ID starting with 28f04748c2990fc6b8c23ed57f6df2026b60a46591783d8fe22d8c2e156df2f0 not found: ID does not exist" Oct 30 22:25:14.473806 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.473568 6006 generic.go:356] "Generic (PLEG): container finished" podID="a852f49d-5ed1-472d-8fb0-f811bbff7e4f" containerID="1070d264a2421a18dfb291e0a9ba378427eef18cc6a07b8df53c4badd4b28a2e" exitCode=0 Oct 30 22:25:14.473806 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.473755 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" event={"ID":"a852f49d-5ed1-472d-8fb0-f811bbff7e4f","Type":"ContainerDied","Data":"1070d264a2421a18dfb291e0a9ba378427eef18cc6a07b8df53c4badd4b28a2e"} Oct 30 22:25:14.484341 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.479468 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:25:14.518038 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.517887 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.518038 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.517921 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.518038 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.517937 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.518038 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.517949 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.518038 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.517962 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.518038 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.517976 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.518038 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.517988 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ba26ec-c30a-46f7-8b3d-d6ef88849246-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.518038 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.518000 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ba26ec-c30a-46f7-8b3d-d6ef88849246-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.518038 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.518012 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4h8l\" (UniqueName: \"kubernetes.io/projected/67ba26ec-c30a-46f7-8b3d-d6ef88849246-kube-api-access-z4h8l\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.525767 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.525689 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:25:14.630516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.628667 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/492acb10-fd86-4613-a1fc-68a0701b7846-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492acb10-fd86-4613-a1fc-68a0701b7846-istiod-ca-cert\") pod \"492acb10-fd86-4613-a1fc-68a0701b7846\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " Oct 30 22:25:14.630516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.628730 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-workload-certs\") pod \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " Oct 30 22:25:14.630516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.628787 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-workload-socket\") pod \"492acb10-fd86-4613-a1fc-68a0701b7846\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " Oct 30 22:25:14.630516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.628871 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-credential-socket\") pod \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " Oct 30 22:25:14.630516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.628936 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/492acb10-fd86-4613-a1fc-68a0701b7846-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492acb10-fd86-4613-a1fc-68a0701b7846-istio-podinfo\") pod \"492acb10-fd86-4613-a1fc-68a0701b7846\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " Oct 30 22:25:14.630516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.628969 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-istio-envoy\") pod \"492acb10-fd86-4613-a1fc-68a0701b7846\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " Oct 30 22:25:14.630516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.629004 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istiod-ca-cert\") pod \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " Oct 30 22:25:14.630516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.629050 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-podinfo\") pod \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " Oct 30 22:25:14.630516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.629132 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-data\") pod \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " Oct 30 22:25:14.630516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.629186 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/492acb10-fd86-4613-a1fc-68a0701b7846-istio-token\" (UniqueName: \"kubernetes.io/projected/492acb10-fd86-4613-a1fc-68a0701b7846-istio-token\") pod \"492acb10-fd86-4613-a1fc-68a0701b7846\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " Oct 30 22:25:14.630516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.629238 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-credential-socket\") pod \"492acb10-fd86-4613-a1fc-68a0701b7846\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " Oct 30 22:25:14.630516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.629317 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-kube-api-access-w7xh9\" (UniqueName: \"kubernetes.io/projected/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-kube-api-access-w7xh9\") pod \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " Oct 30 22:25:14.630516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.629371 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-envoy\") pod \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " Oct 30 22:25:14.630516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.629402 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-workload-socket\") pod \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " Oct 30 22:25:14.630516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.629452 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-istio-data\") pod \"492acb10-fd86-4613-a1fc-68a0701b7846\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " Oct 30 22:25:14.630516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.629502 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-workload-certs\") pod \"492acb10-fd86-4613-a1fc-68a0701b7846\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " Oct 30 22:25:14.630516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.629569 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-token\" (UniqueName: \"kubernetes.io/projected/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-token\") pod \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\" (UID: \"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6\") " Oct 30 22:25:14.630516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.629625 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/492acb10-fd86-4613-a1fc-68a0701b7846-kube-api-access-9z7kh\" (UniqueName: \"kubernetes.io/projected/492acb10-fd86-4613-a1fc-68a0701b7846-kube-api-access-9z7kh\") pod \"492acb10-fd86-4613-a1fc-68a0701b7846\" (UID: \"492acb10-fd86-4613-a1fc-68a0701b7846\") " Oct 30 22:25:14.631859 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.631553 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-data" pod "4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6" (UID: "4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.631913 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.631892 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/492acb10-fd86-4613-a1fc-68a0701b7846-istiod-ca-cert" pod "492acb10-fd86-4613-a1fc-68a0701b7846" (UID: "492acb10-fd86-4613-a1fc-68a0701b7846"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:14.632876 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.632163 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-workload-certs" pod "4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6" (UID: "4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.632876 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.632333 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-workload-socket" pod "492acb10-fd86-4613-a1fc-68a0701b7846" (UID: "492acb10-fd86-4613-a1fc-68a0701b7846"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.632876 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.632758 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-istio-data" pod "492acb10-fd86-4613-a1fc-68a0701b7846" (UID: "492acb10-fd86-4613-a1fc-68a0701b7846"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.633203 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.633105 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-workload-certs" pod "492acb10-fd86-4613-a1fc-68a0701b7846" (UID: "492acb10-fd86-4613-a1fc-68a0701b7846"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.634884 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.634127 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-workload-socket" pod "4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6" (UID: "4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.635274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.635149 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istiod-ca-cert" pod "4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6" (UID: "4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:14.635697 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.635412 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492acb10-fd86-4613-a1fc-68a0701b7846-kube-api-access-9z7kh" pod "492acb10-fd86-4613-a1fc-68a0701b7846" (UID: "492acb10-fd86-4613-a1fc-68a0701b7846"). InnerVolumeSpecName "kube-api-access-9z7kh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:14.635697 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.635596 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-credential-socket" pod "4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6" (UID: "4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.635850 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.635808 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-credential-socket" pod "492acb10-fd86-4613-a1fc-68a0701b7846" (UID: "492acb10-fd86-4613-a1fc-68a0701b7846"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.638379 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.638344 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-istio-envoy" pod "492acb10-fd86-4613-a1fc-68a0701b7846" (UID: "492acb10-fd86-4613-a1fc-68a0701b7846"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.663363 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.640906 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492acb10-fd86-4613-a1fc-68a0701b7846-istio-token" pod "492acb10-fd86-4613-a1fc-68a0701b7846" (UID: "492acb10-fd86-4613-a1fc-68a0701b7846"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:14.664603 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.664547 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-kube-api-access-w7xh9" pod "4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6" (UID: "4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6"). InnerVolumeSpecName "kube-api-access-w7xh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:14.665017 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.664986 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/492acb10-fd86-4613-a1fc-68a0701b7846-istio-podinfo" pod "492acb10-fd86-4613-a1fc-68a0701b7846" (UID: "492acb10-fd86-4613-a1fc-68a0701b7846"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:25:14.676354 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.676310 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-podinfo" pod "4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6" (UID: "4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:25:14.685145 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.683570 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-envoy" pod "4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6" (UID: "4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.687335 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.687222 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-token" pod "4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6" (UID: "4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:14.701892 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.701576 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:25:14.706909 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.706880 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:25:14.722536 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.714813 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:25:14.729065 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.729038 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:25:14.731891 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.731860 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492acb10-fd86-4613-a1fc-68a0701b7846-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.732022 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.731893 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.732022 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.731930 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.732022 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.731943 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.732022 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.731995 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492acb10-fd86-4613-a1fc-68a0701b7846-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.732022 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.732016 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.732208 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.732030 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.732208 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.732044 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.732208 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.732057 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.732208 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.732070 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492acb10-fd86-4613-a1fc-68a0701b7846-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.732208 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.732083 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.732208 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.732107 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7xh9\" (UniqueName: \"kubernetes.io/projected/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-kube-api-access-w7xh9\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.732208 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.732121 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.732208 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.732134 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.732208 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.732147 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.732208 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.732161 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492acb10-fd86-4613-a1fc-68a0701b7846-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.732208 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.732173 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.732208 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.732188 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z7kh\" (UniqueName: \"kubernetes.io/projected/492acb10-fd86-4613-a1fc-68a0701b7846-kube-api-access-9z7kh\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.735569 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.735538 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:25:14.773122 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.773090 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn"] Oct 30 22:25:14.825978 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.825943 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-d9zmn"] Oct 30 22:25:14.832700 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.832661 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-envoy\") pod \"46a061d3-8105-4260-9bf1-4fc3a5a09343\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " Oct 30 22:25:14.832884 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.832715 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf391b9e-a66a-4b60-9c45-8025b5155829-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf391b9e-a66a-4b60-9c45-8025b5155829-istiod-ca-cert\") pod \"bf391b9e-a66a-4b60-9c45-8025b5155829\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " Oct 30 22:25:14.832884 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.832748 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-data\") pod \"46a061d3-8105-4260-9bf1-4fc3a5a09343\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " Oct 30 22:25:14.832884 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.832791 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-workload-socket\") pod \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " Oct 30 22:25:14.832884 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.832852 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-token\" (UniqueName: \"kubernetes.io/projected/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-token\") pod \"46a061d3-8105-4260-9bf1-4fc3a5a09343\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " Oct 30 22:25:14.833115 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.832897 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-envoy\") pod \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " Oct 30 22:25:14.833115 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.832957 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-workload-socket\") pod \"46a061d3-8105-4260-9bf1-4fc3a5a09343\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " Oct 30 22:25:14.833115 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.832995 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-workload-socket\") pod \"2cb25e8f-893d-486f-aed7-922d9804d9c9\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " Oct 30 22:25:14.833115 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.833033 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-kube-api-access-zbtlw\" (UniqueName: \"kubernetes.io/projected/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-kube-api-access-zbtlw\") pod \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " Oct 30 22:25:14.833115 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.833093 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-data\") pod \"bf391b9e-a66a-4b60-9c45-8025b5155829\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " Oct 30 22:25:14.833419 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.833114 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-workload-socket" pod "5aa0ffc2-0a81-47ff-83c5-751eff96ac2d" (UID: "5aa0ffc2-0a81-47ff-83c5-751eff96ac2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.833419 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.833125 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-podinfo\") pod \"2cb25e8f-893d-486f-aed7-922d9804d9c9\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " Oct 30 22:25:14.833419 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.833191 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf391b9e-a66a-4b60-9c45-8025b5155829-istiod-ca-cert" pod "bf391b9e-a66a-4b60-9c45-8025b5155829" (UID: "bf391b9e-a66a-4b60-9c45-8025b5155829"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:14.833567 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.833495 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-data" pod "46a061d3-8105-4260-9bf1-4fc3a5a09343" (UID: "46a061d3-8105-4260-9bf1-4fc3a5a09343"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.833779 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.833715 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-data\") pod \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " Oct 30 22:25:14.833779 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.833741 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-workload-socket" pod "2cb25e8f-893d-486f-aed7-922d9804d9c9" (UID: "2cb25e8f-893d-486f-aed7-922d9804d9c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.833934 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.833782 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-credential-socket\") pod \"2cb25e8f-893d-486f-aed7-922d9804d9c9\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " Oct 30 22:25:14.833934 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.833819 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46a061d3-8105-4260-9bf1-4fc3a5a09343-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46a061d3-8105-4260-9bf1-4fc3a5a09343-istiod-ca-cert\") pod \"46a061d3-8105-4260-9bf1-4fc3a5a09343\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " Oct 30 22:25:14.833934 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.833842 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-workload-socket" pod "46a061d3-8105-4260-9bf1-4fc3a5a09343" (UID: "46a061d3-8105-4260-9bf1-4fc3a5a09343"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.833934 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.833862 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-data" pod "bf391b9e-a66a-4b60-9c45-8025b5155829" (UID: "bf391b9e-a66a-4b60-9c45-8025b5155829"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.833934 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.833868 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-workload-certs\") pod \"46a061d3-8105-4260-9bf1-4fc3a5a09343\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " Oct 30 22:25:14.833934 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.833923 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-token\" (UniqueName: \"kubernetes.io/projected/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-token\") pod \"2cb25e8f-893d-486f-aed7-922d9804d9c9\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " Oct 30 22:25:14.834229 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.833961 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cb25e8f-893d-486f-aed7-922d9804d9c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb25e8f-893d-486f-aed7-922d9804d9c9-istiod-ca-cert\") pod \"2cb25e8f-893d-486f-aed7-922d9804d9c9\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " Oct 30 22:25:14.834229 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.834043 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-envoy\") pod \"bf391b9e-a66a-4b60-9c45-8025b5155829\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " Oct 30 22:25:14.834229 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.834045 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-data" pod "5aa0ffc2-0a81-47ff-83c5-751eff96ac2d" (UID: "5aa0ffc2-0a81-47ff-83c5-751eff96ac2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.834229 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.834090 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-kube-api-access-ch8nj\" (UniqueName: \"kubernetes.io/projected/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-kube-api-access-ch8nj\") pod \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " Oct 30 22:25:14.834229 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.834107 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-workload-certs" pod "46a061d3-8105-4260-9bf1-4fc3a5a09343" (UID: "46a061d3-8105-4260-9bf1-4fc3a5a09343"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.834229 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.834139 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istiod-ca-cert\") pod \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " Oct 30 22:25:14.834229 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.834205 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb25e8f-893d-486f-aed7-922d9804d9c9-kube-api-access-j48xn\" (UniqueName: \"kubernetes.io/projected/2cb25e8f-893d-486f-aed7-922d9804d9c9-kube-api-access-j48xn\") pod \"2cb25e8f-893d-486f-aed7-922d9804d9c9\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " Oct 30 22:25:14.834625 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.834232 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-credential-socket" pod "2cb25e8f-893d-486f-aed7-922d9804d9c9" (UID: "2cb25e8f-893d-486f-aed7-922d9804d9c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.834625 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.834234 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-data\") pod \"2cb25e8f-893d-486f-aed7-922d9804d9c9\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " Oct 30 22:25:14.834625 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.834300 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-credential-socket\") pod \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " Oct 30 22:25:14.834625 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.834339 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-credential-socket\") pod \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " Oct 30 22:25:14.834625 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.834368 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-podinfo\") pod \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " Oct 30 22:25:14.834625 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.834429 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-token\" (UniqueName: \"kubernetes.io/projected/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-token\") pod \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " Oct 30 22:25:14.834625 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.834460 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-data" pod "2cb25e8f-893d-486f-aed7-922d9804d9c9" (UID: "2cb25e8f-893d-486f-aed7-922d9804d9c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.834625 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.834464 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-token\" (UniqueName: \"kubernetes.io/projected/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-token\") pod \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " Oct 30 22:25:14.834625 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.834551 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-workload-certs\") pod \"bf391b9e-a66a-4b60-9c45-8025b5155829\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " Oct 30 22:25:14.834625 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.834583 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-podinfo\") pod \"bf391b9e-a66a-4b60-9c45-8025b5155829\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " Oct 30 22:25:14.835117 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.834654 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-credential-socket\") pod \"46a061d3-8105-4260-9bf1-4fc3a5a09343\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " Oct 30 22:25:14.835117 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.834695 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-workload-certs\") pod \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " Oct 30 22:25:14.835117 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.834705 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46a061d3-8105-4260-9bf1-4fc3a5a09343-istiod-ca-cert" pod "46a061d3-8105-4260-9bf1-4fc3a5a09343" (UID: "46a061d3-8105-4260-9bf1-4fc3a5a09343"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:14.835117 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.834737 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46a061d3-8105-4260-9bf1-4fc3a5a09343-kube-api-access-tnzqg\" (UniqueName: \"kubernetes.io/projected/46a061d3-8105-4260-9bf1-4fc3a5a09343-kube-api-access-tnzqg\") pod \"46a061d3-8105-4260-9bf1-4fc3a5a09343\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " Oct 30 22:25:14.835117 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.834785 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-podinfo\") pod \"46a061d3-8105-4260-9bf1-4fc3a5a09343\" (UID: \"46a061d3-8105-4260-9bf1-4fc3a5a09343\") " Oct 30 22:25:14.835117 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.834827 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-envoy\") pod \"2cb25e8f-893d-486f-aed7-922d9804d9c9\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " Oct 30 22:25:14.835117 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.834881 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-workload-socket\") pod \"bf391b9e-a66a-4b60-9c45-8025b5155829\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " Oct 30 22:25:14.835117 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.834913 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-credential-socket\") pod \"bf391b9e-a66a-4b60-9c45-8025b5155829\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " Oct 30 22:25:14.835117 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.834990 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-envoy\") pod \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " Oct 30 22:25:14.835117 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.835086 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-podinfo\") pod \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " Oct 30 22:25:14.835657 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.835138 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-workload-certs\") pod \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\" (UID: \"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d\") " Oct 30 22:25:14.835657 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.835170 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-workload-socket\") pod \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " Oct 30 22:25:14.835657 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.835206 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf391b9e-a66a-4b60-9c45-8025b5155829-kube-api-access-scs62\" (UniqueName: \"kubernetes.io/projected/bf391b9e-a66a-4b60-9c45-8025b5155829-kube-api-access-scs62\") pod \"bf391b9e-a66a-4b60-9c45-8025b5155829\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " Oct 30 22:25:14.835657 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.835221 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-credential-socket" pod "5aa0ffc2-0a81-47ff-83c5-751eff96ac2d" (UID: "5aa0ffc2-0a81-47ff-83c5-751eff96ac2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.835657 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.835284 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istiod-ca-cert\") pod \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " Oct 30 22:25:14.835657 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.835335 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-token\" (UniqueName: \"kubernetes.io/projected/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-token\") pod \"bf391b9e-a66a-4b60-9c45-8025b5155829\" (UID: \"bf391b9e-a66a-4b60-9c45-8025b5155829\") " Oct 30 22:25:14.835657 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.835378 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-workload-certs\") pod \"2cb25e8f-893d-486f-aed7-922d9804d9c9\" (UID: \"2cb25e8f-893d-486f-aed7-922d9804d9c9\") " Oct 30 22:25:14.835657 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.835406 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-data\") pod \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\" (UID: \"a852f49d-5ed1-472d-8fb0-f811bbff7e4f\") " Oct 30 22:25:14.835657 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.835617 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-credential-socket" pod "a852f49d-5ed1-472d-8fb0-f811bbff7e4f" (UID: "a852f49d-5ed1-472d-8fb0-f811bbff7e4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.836084 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.835809 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-data" pod "a852f49d-5ed1-472d-8fb0-f811bbff7e4f" (UID: "a852f49d-5ed1-472d-8fb0-f811bbff7e4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.836427 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.836389 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.836427 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.836419 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf391b9e-a66a-4b60-9c45-8025b5155829-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.836624 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.836435 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.836624 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.836453 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.836624 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.836467 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.836624 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.836481 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.836624 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.836494 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.836624 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.836507 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.836624 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.836521 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.836624 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.836536 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46a061d3-8105-4260-9bf1-4fc3a5a09343-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.836624 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.836550 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.836624 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.836564 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.836624 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.836578 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.836624 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.836592 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.837693 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.836803 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-workload-socket" pod "a852f49d-5ed1-472d-8fb0-f811bbff7e4f" (UID: "a852f49d-5ed1-472d-8fb0-f811bbff7e4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.838562 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.837695 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-workload-certs" pod "5aa0ffc2-0a81-47ff-83c5-751eff96ac2d" (UID: "5aa0ffc2-0a81-47ff-83c5-751eff96ac2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.838562 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.838049 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-workload-socket" pod "bf391b9e-a66a-4b60-9c45-8025b5155829" (UID: "bf391b9e-a66a-4b60-9c45-8025b5155829"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.838562 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.838147 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-workload-certs" pod "2cb25e8f-893d-486f-aed7-922d9804d9c9" (UID: "2cb25e8f-893d-486f-aed7-922d9804d9c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.838562 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.838225 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-credential-socket" pod "bf391b9e-a66a-4b60-9c45-8025b5155829" (UID: "bf391b9e-a66a-4b60-9c45-8025b5155829"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.840544 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.840173 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-workload-certs" pod "bf391b9e-a66a-4b60-9c45-8025b5155829" (UID: "bf391b9e-a66a-4b60-9c45-8025b5155829"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.843667 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.841906 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb25e8f-893d-486f-aed7-922d9804d9c9-istiod-ca-cert" pod "2cb25e8f-893d-486f-aed7-922d9804d9c9" (UID: "2cb25e8f-893d-486f-aed7-922d9804d9c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:14.843667 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.842533 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-workload-certs" pod "a852f49d-5ed1-472d-8fb0-f811bbff7e4f" (UID: "a852f49d-5ed1-472d-8fb0-f811bbff7e4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.843667 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.843148 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istiod-ca-cert" pod "5aa0ffc2-0a81-47ff-83c5-751eff96ac2d" (UID: "5aa0ffc2-0a81-47ff-83c5-751eff96ac2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:14.843667 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.843579 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istiod-ca-cert" pod "a852f49d-5ed1-472d-8fb0-f811bbff7e4f" (UID: "a852f49d-5ed1-472d-8fb0-f811bbff7e4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:14.843667 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.843645 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-credential-socket" pod "46a061d3-8105-4260-9bf1-4fc3a5a09343" (UID: "46a061d3-8105-4260-9bf1-4fc3a5a09343"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.845854 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.845818 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-token" pod "46a061d3-8105-4260-9bf1-4fc3a5a09343" (UID: "46a061d3-8105-4260-9bf1-4fc3a5a09343"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:14.846683 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.846637 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-envoy" pod "2cb25e8f-893d-486f-aed7-922d9804d9c9" (UID: "2cb25e8f-893d-486f-aed7-922d9804d9c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.846777 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.846706 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf391b9e-a66a-4b60-9c45-8025b5155829-kube-api-access-scs62" pod "bf391b9e-a66a-4b60-9c45-8025b5155829" (UID: "bf391b9e-a66a-4b60-9c45-8025b5155829"). InnerVolumeSpecName "kube-api-access-scs62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:14.846911 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.846852 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-podinfo" pod "46a061d3-8105-4260-9bf1-4fc3a5a09343" (UID: "46a061d3-8105-4260-9bf1-4fc3a5a09343"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:25:14.846972 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.846957 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-token" pod "5aa0ffc2-0a81-47ff-83c5-751eff96ac2d" (UID: "5aa0ffc2-0a81-47ff-83c5-751eff96ac2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:14.847136 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.847108 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-envoy" pod "46a061d3-8105-4260-9bf1-4fc3a5a09343" (UID: "46a061d3-8105-4260-9bf1-4fc3a5a09343"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.847136 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.847119 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-envoy" pod "a852f49d-5ed1-472d-8fb0-f811bbff7e4f" (UID: "a852f49d-5ed1-472d-8fb0-f811bbff7e4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.847136 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.847122 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-kube-api-access-zbtlw" pod "a852f49d-5ed1-472d-8fb0-f811bbff7e4f" (UID: "a852f49d-5ed1-472d-8fb0-f811bbff7e4f"). InnerVolumeSpecName "kube-api-access-zbtlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:14.851229 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.851197 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-envoy" pod "5aa0ffc2-0a81-47ff-83c5-751eff96ac2d" (UID: "5aa0ffc2-0a81-47ff-83c5-751eff96ac2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.851520 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.851480 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-podinfo" pod "a852f49d-5ed1-472d-8fb0-f811bbff7e4f" (UID: "a852f49d-5ed1-472d-8fb0-f811bbff7e4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:25:14.851909 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.851776 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-podinfo" pod "5aa0ffc2-0a81-47ff-83c5-751eff96ac2d" (UID: "5aa0ffc2-0a81-47ff-83c5-751eff96ac2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:25:14.851909 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.851896 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb25e8f-893d-486f-aed7-922d9804d9c9-kube-api-access-j48xn" pod "2cb25e8f-893d-486f-aed7-922d9804d9c9" (UID: "2cb25e8f-893d-486f-aed7-922d9804d9c9"). InnerVolumeSpecName "kube-api-access-j48xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:14.852092 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.851923 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-kube-api-access-ch8nj" pod "5aa0ffc2-0a81-47ff-83c5-751eff96ac2d" (UID: "5aa0ffc2-0a81-47ff-83c5-751eff96ac2d"). InnerVolumeSpecName "kube-api-access-ch8nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:14.852386 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.852360 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a061d3-8105-4260-9bf1-4fc3a5a09343-kube-api-access-tnzqg" pod "46a061d3-8105-4260-9bf1-4fc3a5a09343" (UID: "46a061d3-8105-4260-9bf1-4fc3a5a09343"). InnerVolumeSpecName "kube-api-access-tnzqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:14.852614 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.852579 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-podinfo" pod "bf391b9e-a66a-4b60-9c45-8025b5155829" (UID: "bf391b9e-a66a-4b60-9c45-8025b5155829"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:25:14.852703 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.852612 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-podinfo" pod "2cb25e8f-893d-486f-aed7-922d9804d9c9" (UID: "2cb25e8f-893d-486f-aed7-922d9804d9c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:25:14.852972 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.852944 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-token" pod "a852f49d-5ed1-472d-8fb0-f811bbff7e4f" (UID: "a852f49d-5ed1-472d-8fb0-f811bbff7e4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:14.853075 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.852967 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-token" pod "2cb25e8f-893d-486f-aed7-922d9804d9c9" (UID: "2cb25e8f-893d-486f-aed7-922d9804d9c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:14.853138 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.853105 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-envoy" pod "bf391b9e-a66a-4b60-9c45-8025b5155829" (UID: "bf391b9e-a66a-4b60-9c45-8025b5155829"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.853415 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.853391 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-token" pod "bf391b9e-a66a-4b60-9c45-8025b5155829" (UID: "bf391b9e-a66a-4b60-9c45-8025b5155829"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:14.937891 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.937802 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.937891 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.937847 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.937891 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.937863 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.937891 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.937879 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.937891 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.937892 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.938165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.937907 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbtlw\" (UniqueName: \"kubernetes.io/projected/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-kube-api-access-zbtlw\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.938165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.937922 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.938165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.937938 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.938165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.937952 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb25e8f-893d-486f-aed7-922d9804d9c9-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.938165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.937965 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.938165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.937981 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch8nj\" (UniqueName: \"kubernetes.io/projected/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-kube-api-access-ch8nj\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.938165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.937995 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.938165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.938009 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j48xn\" (UniqueName: \"kubernetes.io/projected/2cb25e8f-893d-486f-aed7-922d9804d9c9-kube-api-access-j48xn\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.938165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.938023 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.938165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.938036 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.938165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.938049 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.938165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.938062 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.938165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.938075 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf391b9e-a66a-4b60-9c45-8025b5155829-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.938165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.938090 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a061d3-8105-4260-9bf1-4fc3a5a09343-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.938165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.938103 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.938165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.938117 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnzqg\" (UniqueName: \"kubernetes.io/projected/46a061d3-8105-4260-9bf1-4fc3a5a09343-kube-api-access-tnzqg\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.938165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.938132 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46a061d3-8105-4260-9bf1-4fc3a5a09343-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.938165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.938145 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb25e8f-893d-486f-aed7-922d9804d9c9-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.938165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.938159 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.938737 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.938172 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf391b9e-a66a-4b60-9c45-8025b5155829-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.938737 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.938187 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.938737 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.938201 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.938737 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.938214 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.938737 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.938228 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.938737 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.938242 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scs62\" (UniqueName: \"kubernetes.io/projected/bf391b9e-a66a-4b60-9c45-8025b5155829-kube-api-access-scs62\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.938737 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:14.938271 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a852f49d-5ed1-472d-8fb0-f811bbff7e4f-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:15.484347 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.484318 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" Oct 30 22:25:15.484552 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.484309 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc" event={"ID":"46a061d3-8105-4260-9bf1-4fc3a5a09343","Type":"ContainerDied","Data":"0161029169b6a4955eae40d5fe6f5d7f9e6e7639d242ef776697863b83c0fa3d"} Oct 30 22:25:15.484552 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.484462 6006 scope.go:122] "RemoveContainer" containerID="fbc2fbcbea5902da8911c902b7dc076c87011d543a001f4730bd3588bacc4790" Oct 30 22:25:15.486227 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.486199 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" event={"ID":"bf391b9e-a66a-4b60-9c45-8025b5155829","Type":"ContainerDied","Data":"65741f23f5cc5a765238efca3abd00fbaf97898579d6a6a6ad6c024480a4b5f9"} Oct 30 22:25:15.486356 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.486216 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt" Oct 30 22:25:15.488126 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.488079 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" event={"ID":"492acb10-fd86-4613-a1fc-68a0701b7846","Type":"ContainerDied","Data":"37b04c9bd43cc83beb59eccad432fdfdef7cb786ebd327b2a270733ee575423b"} Oct 30 22:25:15.488235 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.488185 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf" Oct 30 22:25:15.496438 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.496330 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" Oct 30 22:25:15.496719 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.496330 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p" event={"ID":"5aa0ffc2-0a81-47ff-83c5-751eff96ac2d","Type":"ContainerDied","Data":"85f4b62f2ca41caaf2bd72db48542272e69b430ff250ab37e51c6cdd3bac8f2d"} Oct 30 22:25:15.497624 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.497531 6006 scope.go:122] "RemoveContainer" containerID="a15439709ea391e6878e0220c6ba1ea463681ea21e3a3372672298292b75475e" Oct 30 22:25:15.498303 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.498147 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" Oct 30 22:25:15.498303 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.498158 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv" event={"ID":"4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6","Type":"ContainerDied","Data":"c9cc07407ee84b298be6e208f0ba620117e5883165860c1140cac089b0ed6333"} Oct 30 22:25:15.500220 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.500185 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" event={"ID":"2cb25e8f-893d-486f-aed7-922d9804d9c9","Type":"ContainerDied","Data":"c48dc50022b0fb3b46780b81a6b625bd26e3b7e4d1eba9bf964e2eb05baa616a"} Oct 30 22:25:15.500413 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.500199 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6" Oct 30 22:25:15.507049 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.507022 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" event={"ID":"a852f49d-5ed1-472d-8fb0-f811bbff7e4f","Type":"ContainerDied","Data":"dab45a25adab0b275b28b4948b6f74e40f5ad6bfe7a134e1a7c89eac48580669"} Oct 30 22:25:15.507198 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.507154 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2" Oct 30 22:25:15.514364 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.514338 6006 scope.go:122] "RemoveContainer" containerID="50c3fe8a4021575af554f8504c4e041e1c32f8e77051a3555055129cf20d36e3" Oct 30 22:25:15.525985 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.525957 6006 scope.go:122] "RemoveContainer" containerID="69b2526d6aa09976a97f48bd39c2752b245fc511fe5ddbfb1b8126795ebf3f2f" Oct 30 22:25:15.536304 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.536285 6006 scope.go:122] "RemoveContainer" containerID="1c8c6a047107f5731955e18abb5ab34dee947652edcb3fa9e97321ac5105ed23" Oct 30 22:25:15.552659 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.552144 6006 scope.go:122] "RemoveContainer" containerID="b9e4c73a51ed3bae59bcc8585870682149154c76278a5a0ff30624a82693e525" Oct 30 22:25:15.565994 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.565962 6006 scope.go:122] "RemoveContainer" containerID="1070d264a2421a18dfb291e0a9ba378427eef18cc6a07b8df53c4badd4b28a2e" Oct 30 22:25:15.610434 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.610396 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67ba26ec-c30a-46f7-8b3d-d6ef88849246" path="/var/lib/kubelet/pods/67ba26ec-c30a-46f7-8b3d-d6ef88849246/volumes" Oct 30 22:25:15.678496 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.678450 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p"] Oct 30 22:25:15.729185 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.729146 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-5rz7p"] Oct 30 22:25:15.807925 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.807888 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2"] Oct 30 22:25:15.858184 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.858135 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-qj6l2"] Oct 30 22:25:15.990727 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:15.990692 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6"] Oct 30 22:25:16.040140 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:16.040084 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-htdx6"] Oct 30 22:25:16.113134 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:16.113020 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt"] Oct 30 22:25:16.163719 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:16.163681 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-pf8wt"] Oct 30 22:25:16.234239 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:16.234204 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf"] Oct 30 22:25:16.286554 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:16.286519 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-jsmwf"] Oct 30 22:25:16.358389 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:16.358353 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv"] Oct 30 22:25:16.409068 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:16.408980 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-xltkv"] Oct 30 22:25:16.478866 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:16.478831 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc"] Oct 30 22:25:16.530519 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:16.530476 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-ch8nc"] Oct 30 22:25:17.608166 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:17.608123 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cb25e8f-893d-486f-aed7-922d9804d9c9" path="/var/lib/kubelet/pods/2cb25e8f-893d-486f-aed7-922d9804d9c9/volumes" Oct 30 22:25:17.608878 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:17.608856 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46a061d3-8105-4260-9bf1-4fc3a5a09343" path="/var/lib/kubelet/pods/46a061d3-8105-4260-9bf1-4fc3a5a09343/volumes" Oct 30 22:25:17.609416 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:17.609397 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="492acb10-fd86-4613-a1fc-68a0701b7846" path="/var/lib/kubelet/pods/492acb10-fd86-4613-a1fc-68a0701b7846/volumes" Oct 30 22:25:17.610209 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:17.610187 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6" path="/var/lib/kubelet/pods/4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6/volumes" Oct 30 22:25:17.610717 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:17.610698 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aa0ffc2-0a81-47ff-83c5-751eff96ac2d" path="/var/lib/kubelet/pods/5aa0ffc2-0a81-47ff-83c5-751eff96ac2d/volumes" Oct 30 22:25:17.611193 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:17.611173 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a852f49d-5ed1-472d-8fb0-f811bbff7e4f" path="/var/lib/kubelet/pods/a852f49d-5ed1-472d-8fb0-f811bbff7e4f/volumes" Oct 30 22:25:17.611953 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:17.611932 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf391b9e-a66a-4b60-9c45-8025b5155829" path="/var/lib/kubelet/pods/bf391b9e-a66a-4b60-9c45-8025b5155829/volumes" Oct 30 22:25:18.721845 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.721801 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp"] Oct 30 22:25:18.722422 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.722403 6006 cpu_manager.go:414] "containerMap: removing container" podUID="2cb25e8f-893d-486f-aed7-922d9804d9c9" containerName="istio-proxy" Oct 30 22:25:18.722422 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.722417 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb25e8f-893d-486f-aed7-922d9804d9c9" containerName="istio-proxy" Oct 30 22:25:18.722533 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.722432 6006 cpu_manager.go:414] "containerMap: removing container" podUID="bf391b9e-a66a-4b60-9c45-8025b5155829" containerName="istio-proxy" Oct 30 22:25:18.722533 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.722438 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf391b9e-a66a-4b60-9c45-8025b5155829" containerName="istio-proxy" Oct 30 22:25:18.722533 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.722467 6006 cpu_manager.go:414] "containerMap: removing container" podUID="5aa0ffc2-0a81-47ff-83c5-751eff96ac2d" containerName="istio-proxy" Oct 30 22:25:18.722533 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.722473 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa0ffc2-0a81-47ff-83c5-751eff96ac2d" containerName="istio-proxy" Oct 30 22:25:18.722533 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.722481 6006 cpu_manager.go:414] "containerMap: removing container" podUID="46a061d3-8105-4260-9bf1-4fc3a5a09343" containerName="istio-proxy" Oct 30 22:25:18.722533 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.722486 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46a061d3-8105-4260-9bf1-4fc3a5a09343" containerName="istio-proxy" Oct 30 22:25:18.722533 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.722491 6006 cpu_manager.go:414] "containerMap: removing container" podUID="a852f49d-5ed1-472d-8fb0-f811bbff7e4f" containerName="istio-proxy" Oct 30 22:25:18.722533 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.722496 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a852f49d-5ed1-472d-8fb0-f811bbff7e4f" containerName="istio-proxy" Oct 30 22:25:18.722533 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.722503 6006 cpu_manager.go:414] "containerMap: removing container" podUID="492acb10-fd86-4613-a1fc-68a0701b7846" containerName="istio-proxy" Oct 30 22:25:18.722533 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.722507 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="492acb10-fd86-4613-a1fc-68a0701b7846" containerName="istio-proxy" Oct 30 22:25:18.722533 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.722515 6006 cpu_manager.go:414] "containerMap: removing container" podUID="67ba26ec-c30a-46f7-8b3d-d6ef88849246" containerName="istio-proxy" Oct 30 22:25:18.722533 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.722519 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67ba26ec-c30a-46f7-8b3d-d6ef88849246" containerName="istio-proxy" Oct 30 22:25:18.722533 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.722534 6006 cpu_manager.go:414] "containerMap: removing container" podUID="4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6" containerName="istio-proxy" Oct 30 22:25:18.722533 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.722540 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6" containerName="istio-proxy" Oct 30 22:25:18.723203 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.722630 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa0ffc2-0a81-47ff-83c5-751eff96ac2d" containerName="istio-proxy" Oct 30 22:25:18.723203 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.722639 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="46a061d3-8105-4260-9bf1-4fc3a5a09343" containerName="istio-proxy" Oct 30 22:25:18.723203 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.722646 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="a852f49d-5ed1-472d-8fb0-f811bbff7e4f" containerName="istio-proxy" Oct 30 22:25:18.723203 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.722651 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="492acb10-fd86-4613-a1fc-68a0701b7846" containerName="istio-proxy" Oct 30 22:25:18.723203 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.722657 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="67ba26ec-c30a-46f7-8b3d-d6ef88849246" containerName="istio-proxy" Oct 30 22:25:18.723203 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.722666 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f9cf1a6-b28c-4ec7-8e72-4205ae0982c6" containerName="istio-proxy" Oct 30 22:25:18.723203 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.722675 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb25e8f-893d-486f-aed7-922d9804d9c9" containerName="istio-proxy" Oct 30 22:25:18.723203 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.722680 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf391b9e-a66a-4b60-9c45-8025b5155829" containerName="istio-proxy" Oct 30 22:25:18.726019 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.725996 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:18.789362 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.788402 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d7cdcc-openshift-default-dockercfg-zg92f\"" Oct 30 22:25:18.835132 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.835086 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp"] Oct 30 22:25:18.838544 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.838500 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-data\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:18.838719 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.838553 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istiod-ca-cert\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:18.838719 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.838647 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-token\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:18.838719 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.838674 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-credential-socket\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:18.838902 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.838722 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msk6z\" (UniqueName: \"kubernetes.io/projected/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-kube-api-access-msk6z\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:18.838902 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.838780 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-workload-socket\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:18.838902 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.838804 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-podinfo\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:18.838902 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.838857 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-workload-certs\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:18.839132 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.838981 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-envoy\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:18.940265 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.940203 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-token\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:18.940561 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.940534 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-credential-socket\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:18.941296 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.940704 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msk6z\" (UniqueName: \"kubernetes.io/projected/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-kube-api-access-msk6z\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:18.941511 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.941489 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-workload-socket\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:18.941956 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.941936 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-podinfo\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:18.942161 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.942134 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-workload-certs\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:18.942427 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.942410 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-envoy\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:18.942588 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.942574 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-data\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:18.942693 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.942680 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istiod-ca-cert\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:18.943626 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.943603 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istiod-ca-cert\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:18.945471 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.941895 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-workload-socket\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:18.945582 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.940965 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-credential-socket\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:18.945659 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.944851 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-data\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:18.945743 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.945088 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-workload-certs\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:18.947285 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.947269 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-podinfo\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:18.958852 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:18.958779 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-envoy\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:19.006752 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:19.006610 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-token\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:19.006972 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:19.006813 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msk6z\" (UniqueName: \"kubernetes.io/projected/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-kube-api-access-msk6z\") pod \"e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:19.042756 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:19.042719 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:19.508570 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:25:19.508535 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c79304_5ef5_4fc9_ba42_facc6c16f2b0.slice/crio-b54252bbbdd8bd3bb16d4fd81f37c847d9fb7af69e7ad94470909f2cb1d8225c WatchSource:0}: Error finding container b54252bbbdd8bd3bb16d4fd81f37c847d9fb7af69e7ad94470909f2cb1d8225c: Status 404 returned error can't find the container with id b54252bbbdd8bd3bb16d4fd81f37c847d9fb7af69e7ad94470909f2cb1d8225c Oct 30 22:25:19.511761 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:19.511711 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:25:19.511892 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:19.511811 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:25:19.511892 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:19.511848 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:25:19.512700 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:19.512669 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp"] Oct 30 22:25:19.551958 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:19.551922 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" event={"ID":"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0","Type":"ContainerStarted","Data":"b54252bbbdd8bd3bb16d4fd81f37c847d9fb7af69e7ad94470909f2cb1d8225c"} Oct 30 22:25:20.560964 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:20.560927 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" event={"ID":"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0","Type":"ContainerStarted","Data":"b588614d2050bf7f198d97ec8d7765e9acd0373a689feab40d3cd18ba44d2d47"} Oct 30 22:25:20.751941 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:20.751861 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" podStartSLOduration=2.751838168 podStartE2EDuration="2.751838168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:20.740618817 +0000 UTC m=+2123.292795121" watchObservedRunningTime="2026-10-30 22:25:20.751838168 +0000 UTC m=+2123.304014468" Oct 30 22:25:21.044858 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:21.043910 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:21.053032 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:21.053002 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:21.568527 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:21.568491 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:21.569774 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:21.569743 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:25:23.048282 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.045413 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx"] Oct 30 22:25:23.066275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.063511 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.124289 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.121356 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0b6a3f-openshift-default-dockercfg-7g5w7\"" Oct 30 22:25:23.146430 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.146372 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-workload-certs\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.146805 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.146771 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-credential-socket\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.147079 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.146994 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ec694be-68a9-4227-96ac-cf7daac3af60-istiod-ca-cert\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.154126 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.154088 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-data\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.154462 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.154441 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-workload-socket\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.154656 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.154640 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-token\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.154849 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.154831 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt7dl\" (UniqueName: \"kubernetes.io/projected/0ec694be-68a9-4227-96ac-cf7daac3af60-kube-api-access-mt7dl\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.155059 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.155044 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-podinfo\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.155215 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.155198 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-envoy\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.192758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.192718 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx"] Oct 30 22:25:23.278273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.278097 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ec694be-68a9-4227-96ac-cf7daac3af60-istiod-ca-cert\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.278273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.278198 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-data\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.278539 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.278281 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-workload-socket\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.278539 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.278310 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-token\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.278539 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.278361 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt7dl\" (UniqueName: \"kubernetes.io/projected/0ec694be-68a9-4227-96ac-cf7daac3af60-kube-api-access-mt7dl\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.278539 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.278427 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-podinfo\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.278539 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.278457 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-envoy\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.278791 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.278573 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-workload-certs\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.278791 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.278616 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-credential-socket\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.281835 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.279081 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-credential-socket\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.281835 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.279767 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ec694be-68a9-4227-96ac-cf7daac3af60-istiod-ca-cert\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.281835 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.280019 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-data\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.281835 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.280274 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-workload-socket\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.281835 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.281735 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-workload-certs\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.294154 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.294082 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-envoy\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.294722 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.294701 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-podinfo\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.375092 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.375010 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt7dl\" (UniqueName: \"kubernetes.io/projected/0ec694be-68a9-4227-96ac-cf7daac3af60-kube-api-access-mt7dl\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.382030 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.382000 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-token\") pod \"e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.418955 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.418918 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:23.858956 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:25:23.858905 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec694be_68a9_4227_96ac_cf7daac3af60.slice/crio-316ff7f83b9bbff55fc4e418c16a7feaf2eca15d4da8cc09ad9e0e15cdf967ba WatchSource:0}: Error finding container 316ff7f83b9bbff55fc4e418c16a7feaf2eca15d4da8cc09ad9e0e15cdf967ba: Status 404 returned error can't find the container with id 316ff7f83b9bbff55fc4e418c16a7feaf2eca15d4da8cc09ad9e0e15cdf967ba Oct 30 22:25:23.862073 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.862034 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:25:23.862183 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.862123 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:25:23.862183 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.862170 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:25:23.862874 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:23.862849 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx"] Oct 30 22:25:24.593819 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:24.593719 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" event={"ID":"0ec694be-68a9-4227-96ac-cf7daac3af60","Type":"ContainerStarted","Data":"839dfd2571d546fa1f815a462ff1a8cadf8afee3f65d81846b4a4f7885cf1dcb"} Oct 30 22:25:24.593819 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:24.593786 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" event={"ID":"0ec694be-68a9-4227-96ac-cf7daac3af60","Type":"ContainerStarted","Data":"316ff7f83b9bbff55fc4e418c16a7feaf2eca15d4da8cc09ad9e0e15cdf967ba"} Oct 30 22:25:24.774642 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:24.774592 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" podStartSLOduration=2.774579454 podStartE2EDuration="2.774579454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:24.773048474 +0000 UTC m=+2127.325224774" watchObservedRunningTime="2026-10-30 22:25:24.774579454 +0000 UTC m=+2127.326755753" Oct 30 22:25:25.419352 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:25.419313 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:25.425144 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:25.425119 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:25.604227 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:25.604193 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:25.604786 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:25.604317 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:25:27.500663 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:27.500627 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp"] Oct 30 22:25:27.505284 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:27.505225 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp" Oct 30 22:25:27.564324 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:27.564285 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp"] Oct 30 22:25:27.662275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:27.659830 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d7cdcc-dockercfg-hkf55\"" Oct 30 22:25:27.662275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:27.659930 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d7cdcc\"" Oct 30 22:25:27.674278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:27.666183 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhssd\" (UniqueName: \"kubernetes.io/projected/d0b10d96-ee84-4a97-b5de-20e08008815d-kube-api-access-jhssd\") pod \"payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp\" (UID: \"d0b10d96-ee84-4a97-b5de-20e08008815d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp" Oct 30 22:25:27.738683 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:27.738475 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0b10d96-ee84-4a97-b5de-20e08008815d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp\" (UID: \"d0b10d96-ee84-4a97-b5de-20e08008815d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp" Oct 30 22:25:27.758236 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:27.758157 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz"] Oct 30 22:25:27.772518 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:27.772480 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz"] Oct 30 22:25:27.773827 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:27.773804 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz" Oct 30 22:25:27.841282 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:27.841216 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0b10d96-ee84-4a97-b5de-20e08008815d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp\" (UID: \"d0b10d96-ee84-4a97-b5de-20e08008815d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp" Oct 30 22:25:27.841513 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:27.841497 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhssd\" (UniqueName: \"kubernetes.io/projected/d0b10d96-ee84-4a97-b5de-20e08008815d-kube-api-access-jhssd\") pod \"payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp\" (UID: \"d0b10d96-ee84-4a97-b5de-20e08008815d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp" Oct 30 22:25:27.842186 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:27.842164 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0b10d96-ee84-4a97-b5de-20e08008815d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp\" (UID: \"d0b10d96-ee84-4a97-b5de-20e08008815d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp" Oct 30 22:25:27.904043 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:27.904015 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhssd\" (UniqueName: \"kubernetes.io/projected/d0b10d96-ee84-4a97-b5de-20e08008815d-kube-api-access-jhssd\") pod \"payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp\" (UID: \"d0b10d96-ee84-4a97-b5de-20e08008815d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp" Oct 30 22:25:27.920711 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:25:27.920671 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:25:27.943122 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:27.943082 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rgzn\" (UniqueName: \"kubernetes.io/projected/695df2fa-fb27-4a50-995e-acb44e355440-kube-api-access-5rgzn\") pod \"payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz\" (UID: \"695df2fa-fb27-4a50-995e-acb44e355440\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz" Oct 30 22:25:27.943456 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:27.943426 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695df2fa-fb27-4a50-995e-acb44e355440-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz\" (UID: \"695df2fa-fb27-4a50-995e-acb44e355440\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz" Oct 30 22:25:28.045200 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:28.045158 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rgzn\" (UniqueName: \"kubernetes.io/projected/695df2fa-fb27-4a50-995e-acb44e355440-kube-api-access-5rgzn\") pod \"payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz\" (UID: \"695df2fa-fb27-4a50-995e-acb44e355440\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz" Oct 30 22:25:28.045452 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:28.045368 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695df2fa-fb27-4a50-995e-acb44e355440-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz\" (UID: \"695df2fa-fb27-4a50-995e-acb44e355440\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz" Oct 30 22:25:28.046126 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:28.046100 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695df2fa-fb27-4a50-995e-acb44e355440-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz\" (UID: \"695df2fa-fb27-4a50-995e-acb44e355440\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz" Oct 30 22:25:28.110461 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:28.110428 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rgzn\" (UniqueName: \"kubernetes.io/projected/695df2fa-fb27-4a50-995e-acb44e355440-kube-api-access-5rgzn\") pod \"payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz\" (UID: \"695df2fa-fb27-4a50-995e-acb44e355440\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz" Oct 30 22:25:28.124431 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:28.124395 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp" Oct 30 22:25:28.395834 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:28.395755 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz" Oct 30 22:25:28.523823 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:28.511090 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp"] Oct 30 22:25:28.777767 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:28.777672 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp" event={"ID":"d0b10d96-ee84-4a97-b5de-20e08008815d","Type":"ContainerStarted","Data":"6f38731e9a9354379ba0bc76fb7562d3d1283ccdafe79923a502af1ff01cfc4e"} Oct 30 22:25:28.777767 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:28.777759 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp" event={"ID":"d0b10d96-ee84-4a97-b5de-20e08008815d","Type":"ContainerStarted","Data":"dc5fd9192392cf4d008eab1643b0e0bc9a80e0ea648e3ddf6d4c5db473c472f5"} Oct 30 22:25:28.777958 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:28.777920 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp" Oct 30 22:25:28.785061 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:25:28.785032 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695df2fa_fb27_4a50_995e_acb44e355440.slice/crio-c7b84de1dc000bd843f5db69fc740936148c8a2363cb24287d1b77a3aef6da48 WatchSource:0}: Error finding container c7b84de1dc000bd843f5db69fc740936148c8a2363cb24287d1b77a3aef6da48: Status 404 returned error can't find the container with id c7b84de1dc000bd843f5db69fc740936148c8a2363cb24287d1b77a3aef6da48 Oct 30 22:25:28.787072 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:28.787047 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz"] Oct 30 22:25:28.899306 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:28.899221 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp" podStartSLOduration=1.899201106 podStartE2EDuration="1.899201106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:28.89756372 +0000 UTC m=+2131.449740019" watchObservedRunningTime="2026-10-30 22:25:28.899201106 +0000 UTC m=+2131.451377407" Oct 30 22:25:29.786190 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:29.786145 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz" event={"ID":"695df2fa-fb27-4a50-995e-acb44e355440","Type":"ContainerStarted","Data":"acc438b39afb01ec9ca8cf95158f5add3753a4e25fcd418e94de976e1a47676c"} Oct 30 22:25:29.786749 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:29.786210 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz" event={"ID":"695df2fa-fb27-4a50-995e-acb44e355440","Type":"ContainerStarted","Data":"c7b84de1dc000bd843f5db69fc740936148c8a2363cb24287d1b77a3aef6da48"} Oct 30 22:25:29.786749 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:29.786408 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz" Oct 30 22:25:29.908565 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:29.908507 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz" podStartSLOduration=2.908491749 podStartE2EDuration="2.908491749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:29.906627767 +0000 UTC m=+2132.458804066" watchObservedRunningTime="2026-10-30 22:25:29.908491749 +0000 UTC m=+2132.460668050" Oct 30 22:25:32.459895 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:32.459853 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq"] Oct 30 22:25:32.464798 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:32.464768 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq" Oct 30 22:25:32.571652 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:32.571612 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0b6a3f-dockercfg-cns8g\"" Oct 30 22:25:32.571878 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:32.571691 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0b6a3f\"" Oct 30 22:25:32.574300 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:32.574268 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq"] Oct 30 22:25:32.589457 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:32.589353 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gll26\" (UniqueName: \"kubernetes.io/projected/5ed80968-c4fc-4ba3-8382-97c1d865b5ac-kube-api-access-gll26\") pod \"payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq\" (UID: \"5ed80968-c4fc-4ba3-8382-97c1d865b5ac\") " pod="openshift-ingress/payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq" Oct 30 22:25:32.589640 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:32.589494 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed80968-c4fc-4ba3-8382-97c1d865b5ac-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq\" (UID: \"5ed80968-c4fc-4ba3-8382-97c1d865b5ac\") " pod="openshift-ingress/payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq" Oct 30 22:25:32.691431 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:32.691376 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gll26\" (UniqueName: \"kubernetes.io/projected/5ed80968-c4fc-4ba3-8382-97c1d865b5ac-kube-api-access-gll26\") pod \"payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq\" (UID: \"5ed80968-c4fc-4ba3-8382-97c1d865b5ac\") " pod="openshift-ingress/payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq" Oct 30 22:25:32.691632 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:32.691475 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed80968-c4fc-4ba3-8382-97c1d865b5ac-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq\" (UID: \"5ed80968-c4fc-4ba3-8382-97c1d865b5ac\") " pod="openshift-ingress/payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq" Oct 30 22:25:32.693715 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:32.693683 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed80968-c4fc-4ba3-8382-97c1d865b5ac-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq\" (UID: \"5ed80968-c4fc-4ba3-8382-97c1d865b5ac\") " pod="openshift-ingress/payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq" Oct 30 22:25:32.755395 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:32.755321 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gll26\" (UniqueName: \"kubernetes.io/projected/5ed80968-c4fc-4ba3-8382-97c1d865b5ac-kube-api-access-gll26\") pod \"payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq\" (UID: \"5ed80968-c4fc-4ba3-8382-97c1d865b5ac\") " pod="openshift-ingress/payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq" Oct 30 22:25:32.782045 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:32.782007 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq" Oct 30 22:25:33.177011 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:25:33.176973 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed80968_c4fc_4ba3_8382_97c1d865b5ac.slice/crio-376e6786a798549d5cf0117a27475dbcdcda26ea9e3d33c1fbc5ea35a726099e WatchSource:0}: Error finding container 376e6786a798549d5cf0117a27475dbcdcda26ea9e3d33c1fbc5ea35a726099e: Status 404 returned error can't find the container with id 376e6786a798549d5cf0117a27475dbcdcda26ea9e3d33c1fbc5ea35a726099e Oct 30 22:25:33.179716 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:33.179690 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq"] Oct 30 22:25:33.821787 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:33.821744 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq" event={"ID":"5ed80968-c4fc-4ba3-8382-97c1d865b5ac","Type":"ContainerStarted","Data":"7fcee6c69caf808baea6fc7b46dee2992cc5f9512a34e4522bed263adffc7ffc"} Oct 30 22:25:33.822306 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:33.821811 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq" event={"ID":"5ed80968-c4fc-4ba3-8382-97c1d865b5ac","Type":"ContainerStarted","Data":"376e6786a798549d5cf0117a27475dbcdcda26ea9e3d33c1fbc5ea35a726099e"} Oct 30 22:25:33.822306 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:33.821950 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq" Oct 30 22:25:33.945310 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:33.945208 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq" podStartSLOduration=1.9451936459999999 podStartE2EDuration="1.945193646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:33.943030268 +0000 UTC m=+2136.495206568" watchObservedRunningTime="2026-10-30 22:25:33.945193646 +0000 UTC m=+2136.497369946" Oct 30 22:25:39.281629 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:39.281588 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4"] Oct 30 22:25:39.282183 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:39.281820 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" podUID="efd2de68-03c7-4dc4-8422-a2a7f87aec11" containerName="istio-proxy" containerID="cri-o://88113bb66b9f3aa027181ffce88deb971c12b80208de7cba9fcba6c64685eb55" gracePeriod=30 Oct 30 22:25:39.790285 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:39.790213 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp" Oct 30 22:25:40.797905 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:40.797871 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz" Oct 30 22:25:44.573050 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.572969 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:25:44.665111 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.665068 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-podinfo\") pod \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " Oct 30 22:25:44.665339 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.665164 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-workload-socket\") pod \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " Oct 30 22:25:44.665339 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.665190 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-data\") pod \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " Oct 30 22:25:44.665339 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.665215 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-token\" (UniqueName: \"kubernetes.io/projected/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-token\") pod \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " Oct 30 22:25:44.665518 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.665383 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istiod-ca-cert\") pod \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " Oct 30 22:25:44.665518 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.665417 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-envoy\") pod \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " Oct 30 22:25:44.665518 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.665477 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-credential-socket\") pod \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " Oct 30 22:25:44.665686 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.665577 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-workload-socket" pod "efd2de68-03c7-4dc4-8422-a2a7f87aec11" (UID: "efd2de68-03c7-4dc4-8422-a2a7f87aec11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:44.665686 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.665613 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-data" pod "efd2de68-03c7-4dc4-8422-a2a7f87aec11" (UID: "efd2de68-03c7-4dc4-8422-a2a7f87aec11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:44.665686 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.665630 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-workload-certs\") pod \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " Oct 30 22:25:44.665853 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.665740 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efd2de68-03c7-4dc4-8422-a2a7f87aec11-kube-api-access-kr28t\" (UniqueName: \"kubernetes.io/projected/efd2de68-03c7-4dc4-8422-a2a7f87aec11-kube-api-access-kr28t\") pod \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\" (UID: \"efd2de68-03c7-4dc4-8422-a2a7f87aec11\") " Oct 30 22:25:44.665913 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.665739 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istiod-ca-cert" pod "efd2de68-03c7-4dc4-8422-a2a7f87aec11" (UID: "efd2de68-03c7-4dc4-8422-a2a7f87aec11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:44.665974 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.665820 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-workload-certs" pod "efd2de68-03c7-4dc4-8422-a2a7f87aec11" (UID: "efd2de68-03c7-4dc4-8422-a2a7f87aec11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:44.665974 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.665887 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-credential-socket" pod "efd2de68-03c7-4dc4-8422-a2a7f87aec11" (UID: "efd2de68-03c7-4dc4-8422-a2a7f87aec11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:44.666331 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.666306 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:44.666448 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.666333 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:44.666448 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.666347 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:44.666448 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.666360 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:44.666448 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.666401 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:44.667795 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.667765 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-podinfo" pod "efd2de68-03c7-4dc4-8422-a2a7f87aec11" (UID: "efd2de68-03c7-4dc4-8422-a2a7f87aec11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:25:44.667894 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.667800 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd2de68-03c7-4dc4-8422-a2a7f87aec11-kube-api-access-kr28t" pod "efd2de68-03c7-4dc4-8422-a2a7f87aec11" (UID: "efd2de68-03c7-4dc4-8422-a2a7f87aec11"). InnerVolumeSpecName "kube-api-access-kr28t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:44.668062 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.668042 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-envoy" pod "efd2de68-03c7-4dc4-8422-a2a7f87aec11" (UID: "efd2de68-03c7-4dc4-8422-a2a7f87aec11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:44.668136 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.668065 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-token" pod "efd2de68-03c7-4dc4-8422-a2a7f87aec11" (UID: "efd2de68-03c7-4dc4-8422-a2a7f87aec11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:44.767793 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.767748 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:44.767793 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.767791 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:44.768068 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.767807 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr28t\" (UniqueName: \"kubernetes.io/projected/efd2de68-03c7-4dc4-8422-a2a7f87aec11-kube-api-access-kr28t\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:44.768068 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.767822 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efd2de68-03c7-4dc4-8422-a2a7f87aec11-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:44.833467 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.833377 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq" Oct 30 22:25:44.909298 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.909245 6006 generic.go:356] "Generic (PLEG): container finished" podID="efd2de68-03c7-4dc4-8422-a2a7f87aec11" containerID="88113bb66b9f3aa027181ffce88deb971c12b80208de7cba9fcba6c64685eb55" exitCode=0 Oct 30 22:25:44.909479 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.909290 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" event={"ID":"efd2de68-03c7-4dc4-8422-a2a7f87aec11","Type":"ContainerDied","Data":"88113bb66b9f3aa027181ffce88deb971c12b80208de7cba9fcba6c64685eb55"} Oct 30 22:25:44.909479 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.909311 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" Oct 30 22:25:44.909479 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.909360 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4" event={"ID":"efd2de68-03c7-4dc4-8422-a2a7f87aec11","Type":"ContainerDied","Data":"6646ccf3ca673ac84d9f507533ad518a696c3ad1c8f3c6e5c452075e66b79597"} Oct 30 22:25:44.909479 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.909391 6006 scope.go:122] "RemoveContainer" containerID="88113bb66b9f3aa027181ffce88deb971c12b80208de7cba9fcba6c64685eb55" Oct 30 22:25:44.922235 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.922201 6006 scope.go:122] "RemoveContainer" containerID="88113bb66b9f3aa027181ffce88deb971c12b80208de7cba9fcba6c64685eb55" Oct 30 22:25:44.922669 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:25:44.922645 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88113bb66b9f3aa027181ffce88deb971c12b80208de7cba9fcba6c64685eb55\": container with ID starting with 88113bb66b9f3aa027181ffce88deb971c12b80208de7cba9fcba6c64685eb55 not found: ID does not exist" containerID="88113bb66b9f3aa027181ffce88deb971c12b80208de7cba9fcba6c64685eb55" Oct 30 22:25:44.922755 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:44.922682 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88113bb66b9f3aa027181ffce88deb971c12b80208de7cba9fcba6c64685eb55"} err="failed to get container status \"88113bb66b9f3aa027181ffce88deb971c12b80208de7cba9fcba6c64685eb55\": rpc error: code = NotFound desc = could not find container \"88113bb66b9f3aa027181ffce88deb971c12b80208de7cba9fcba6c64685eb55\": container with ID starting with 88113bb66b9f3aa027181ffce88deb971c12b80208de7cba9fcba6c64685eb55 not found: ID does not exist" Oct 30 22:25:45.075115 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:45.075077 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4"] Oct 30 22:25:45.119183 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:45.119090 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-mmtp4"] Oct 30 22:25:45.608795 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:45.608757 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efd2de68-03c7-4dc4-8422-a2a7f87aec11" path="/var/lib/kubelet/pods/efd2de68-03c7-4dc4-8422-a2a7f87aec11/volumes" Oct 30 22:25:46.297366 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:46.297328 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw"] Oct 30 22:25:46.298027 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:46.298005 6006 cpu_manager.go:414] "containerMap: removing container" podUID="efd2de68-03c7-4dc4-8422-a2a7f87aec11" containerName="istio-proxy" Oct 30 22:25:46.298136 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:46.298030 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efd2de68-03c7-4dc4-8422-a2a7f87aec11" containerName="istio-proxy" Oct 30 22:25:46.298284 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:46.298263 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="efd2de68-03c7-4dc4-8422-a2a7f87aec11" containerName="istio-proxy" Oct 30 22:25:46.304376 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:46.304344 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" Oct 30 22:25:46.359139 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:46.359100 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw"] Oct 30 22:25:46.390537 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:46.390495 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcltk\" (UniqueName: \"kubernetes.io/projected/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-kube-api-access-lcltk\") pod \"maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw\" (UID: \"d255a3e1-7f5a-40a5-98e5-3d481b133f0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" Oct 30 22:25:46.390762 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:46.390653 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-metrics-certs\") pod \"maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw\" (UID: \"d255a3e1-7f5a-40a5-98e5-3d481b133f0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" Oct 30 22:25:46.390846 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:46.390787 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw\" (UID: \"d255a3e1-7f5a-40a5-98e5-3d481b133f0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" Oct 30 22:25:46.390926 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:46.390880 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-maas-api-tls\") pod \"maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw\" (UID: \"d255a3e1-7f5a-40a5-98e5-3d481b133f0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" Oct 30 22:25:46.405834 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:46.405795 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-5637a7-serving-cert\"" Oct 30 22:25:46.492149 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:46.492108 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcltk\" (UniqueName: \"kubernetes.io/projected/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-kube-api-access-lcltk\") pod \"maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw\" (UID: \"d255a3e1-7f5a-40a5-98e5-3d481b133f0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" Oct 30 22:25:46.492380 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:46.492285 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-metrics-certs\") pod \"maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw\" (UID: \"d255a3e1-7f5a-40a5-98e5-3d481b133f0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" Oct 30 22:25:46.492380 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:46.492359 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw\" (UID: \"d255a3e1-7f5a-40a5-98e5-3d481b133f0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" Oct 30 22:25:46.492529 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:46.492433 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-maas-api-tls\") pod \"maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw\" (UID: \"d255a3e1-7f5a-40a5-98e5-3d481b133f0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" Oct 30 22:25:46.492589 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:25:46.492554 6006 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-serving-cert: secret "maas-api-e2e-inf-a-5637a7-serving-cert" not found Oct 30 22:25:46.492641 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:25:46.492630 6006 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-maas-api-tls podName:d255a3e1-7f5a-40a5-98e5-3d481b133f0e nodeName:}" failed. No retries permitted until 2026-10-30 22:25:46.992611953 +0000 UTC m=+2149.544788253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-maas-api-tls") pod "maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" (UID: "d255a3e1-7f5a-40a5-98e5-3d481b133f0e") : secret "maas-api-e2e-inf-a-5637a7-serving-cert" not found Oct 30 22:25:46.493105 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:46.493080 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw\" (UID: \"d255a3e1-7f5a-40a5-98e5-3d481b133f0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" Oct 30 22:25:46.494933 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:46.494910 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-metrics-certs\") pod \"maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw\" (UID: \"d255a3e1-7f5a-40a5-98e5-3d481b133f0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" Oct 30 22:25:46.576717 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:46.575477 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcltk\" (UniqueName: \"kubernetes.io/projected/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-kube-api-access-lcltk\") pod \"maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw\" (UID: \"d255a3e1-7f5a-40a5-98e5-3d481b133f0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" Oct 30 22:25:46.999671 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:46.999571 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-maas-api-tls\") pod \"maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw\" (UID: \"d255a3e1-7f5a-40a5-98e5-3d481b133f0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" Oct 30 22:25:47.002237 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:47.002210 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-maas-api-tls\") pod \"maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw\" (UID: \"d255a3e1-7f5a-40a5-98e5-3d481b133f0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" Oct 30 22:25:47.222203 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:47.222158 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" Oct 30 22:25:47.607556 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:25:47.607517 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd255a3e1_7f5a_40a5_98e5_3d481b133f0e.slice/crio-35d5dc11d15fb05fc1e170e966a0119b749b161263b373147fd8b4ec6cfe13bb WatchSource:0}: Error finding container 35d5dc11d15fb05fc1e170e966a0119b749b161263b373147fd8b4ec6cfe13bb: Status 404 returned error can't find the container with id 35d5dc11d15fb05fc1e170e966a0119b749b161263b373147fd8b4ec6cfe13bb Oct 30 22:25:47.608880 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:47.608854 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw"] Oct 30 22:25:47.940986 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:47.940883 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" event={"ID":"d255a3e1-7f5a-40a5-98e5-3d481b133f0e","Type":"ContainerStarted","Data":"35d5dc11d15fb05fc1e170e966a0119b749b161263b373147fd8b4ec6cfe13bb"} Oct 30 22:25:48.950567 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:48.950465 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" event={"ID":"d255a3e1-7f5a-40a5-98e5-3d481b133f0e","Type":"ContainerStarted","Data":"b3fe7ef4c21797e146bbc9c85d93476878d58bf6b6e30f190ddfcb8ba7003da3"} Oct 30 22:25:48.950951 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:48.950655 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" Oct 30 22:25:49.073434 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:49.073375 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" podStartSLOduration=2.119626267 podStartE2EDuration="3.073358212s" totalImagesPullingTime="953.731945ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:49.069937571 +0000 UTC m=+2151.622113871" watchObservedRunningTime="2026-10-30 22:25:49.073358212 +0000 UTC m=+2151.625534562" Oct 30 22:25:54.963327 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:54.963294 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" Oct 30 22:25:55.353952 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.353911 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6"] Oct 30 22:25:55.359220 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.359192 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.414510 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.414474 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-595fd7d7-openshift-default-dockercfg-prft5\"" Oct 30 22:25:55.443343 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.443302 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6"] Oct 30 22:25:55.451371 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.451325 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dblk2\" (UniqueName: \"kubernetes.io/projected/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-kube-api-access-dblk2\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.451551 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.451400 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-envoy\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.451653 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.451620 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istiod-ca-cert\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.451715 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.451665 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-workload-socket\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.451771 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.451717 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-credential-socket\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.451771 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.451755 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-workload-certs\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.451902 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.451786 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-token\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.451965 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.451935 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-data\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.452023 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.451969 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-podinfo\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.556213 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.556162 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-data\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.556491 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.556469 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-podinfo\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.556683 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.556666 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dblk2\" (UniqueName: \"kubernetes.io/projected/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-kube-api-access-dblk2\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.556803 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.556790 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-envoy\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.556998 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.556985 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istiod-ca-cert\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.557114 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.557102 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-workload-socket\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.557231 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.557218 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-credential-socket\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.557369 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.557354 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-workload-certs\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.557469 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.557457 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-token\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.558767 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.558738 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-data\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.561027 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.560596 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istiod-ca-cert\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.561469 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.561447 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-workload-socket\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.561808 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.561792 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-credential-socket\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.562131 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.562112 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-workload-certs\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.570366 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.570335 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-podinfo\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.572319 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.571584 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-envoy\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.620618 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.620532 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-token\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.628075 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.628031 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dblk2\" (UniqueName: \"kubernetes.io/projected/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-kube-api-access-dblk2\") pod \"e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:55.676810 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:55.676760 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:56.130652 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:56.130616 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6"] Oct 30 22:25:56.133670 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:56.133621 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:25:56.133827 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:56.133780 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:25:56.133889 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:56.133863 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:25:57.018054 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:57.018009 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" event={"ID":"e020ac39-b5d1-4657-9b2b-ce1a06e4c988","Type":"ContainerStarted","Data":"989bcae7013c3d8510aa0f5888ff3ed850c95eaad116bc27c5d9bb33ce63ed01"} Oct 30 22:25:57.018284 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:57.018117 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" event={"ID":"e020ac39-b5d1-4657-9b2b-ce1a06e4c988","Type":"ContainerStarted","Data":"95b349dc0fd962e54459d5b888cc4ed8dfb6fd3fc7d465d2119cd6edc44dcea2"} Oct 30 22:25:57.176765 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:57.176703 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" podStartSLOduration=2.176688415 podStartE2EDuration="2.176688415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:57.175856146 +0000 UTC m=+2159.728032448" watchObservedRunningTime="2026-10-30 22:25:57.176688415 +0000 UTC m=+2159.728864784" Oct 30 22:25:57.677265 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:57.677221 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:57.683358 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:57.683328 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:57.932076 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:25:57.931972 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:25:58.030275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:58.029881 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:58.031237 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:58.031212 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:25:58.719163 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:58.719126 6006 scope.go:122] "RemoveContainer" containerID="50285152caa67245309d0bdfcb823765466773d2db1ecc30fab513689a28a054" Oct 30 22:25:58.731417 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:58.731392 6006 scope.go:122] "RemoveContainer" containerID="29dbcd9bdf8413a1e1fa4f053a60542a79aa9a7c1cfe594b0a9c09eefa2e5adb" Oct 30 22:25:58.742780 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:25:58.742752 6006 scope.go:122] "RemoveContainer" containerID="e2d41ffab8c7590f1a30ca186ebac03140c2bcc3a86b1a34c66fd131145f11b0" Oct 30 22:26:03.736654 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:03.736618 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd"] Oct 30 22:26:03.743163 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:03.743128 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd" Oct 30 22:26:03.798459 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:03.798414 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd"] Oct 30 22:26:03.899041 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:03.899006 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-595fd7d7-dockercfg-qmxxm\"" Oct 30 22:26:03.904181 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:03.904016 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-595fd7d7\"" Oct 30 22:26:04.005843 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:04.005731 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eece1c74-3904-47a7-ab7f-3464ffd6767e-plugins-config-volume\") pod \"payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd\" (UID: \"eece1c74-3904-47a7-ab7f-3464ffd6767e\") " pod="openshift-ingress/payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd" Oct 30 22:26:04.006038 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:04.006002 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frzth\" (UniqueName: \"kubernetes.io/projected/eece1c74-3904-47a7-ab7f-3464ffd6767e-kube-api-access-frzth\") pod \"payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd\" (UID: \"eece1c74-3904-47a7-ab7f-3464ffd6767e\") " pod="openshift-ingress/payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd" Oct 30 22:26:04.107020 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:04.106966 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eece1c74-3904-47a7-ab7f-3464ffd6767e-plugins-config-volume\") pod \"payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd\" (UID: \"eece1c74-3904-47a7-ab7f-3464ffd6767e\") " pod="openshift-ingress/payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd" Oct 30 22:26:04.107200 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:04.107110 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frzth\" (UniqueName: \"kubernetes.io/projected/eece1c74-3904-47a7-ab7f-3464ffd6767e-kube-api-access-frzth\") pod \"payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd\" (UID: \"eece1c74-3904-47a7-ab7f-3464ffd6767e\") " pod="openshift-ingress/payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd" Oct 30 22:26:04.108044 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:04.108016 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eece1c74-3904-47a7-ab7f-3464ffd6767e-plugins-config-volume\") pod \"payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd\" (UID: \"eece1c74-3904-47a7-ab7f-3464ffd6767e\") " pod="openshift-ingress/payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd" Oct 30 22:26:04.170046 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:04.170008 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frzth\" (UniqueName: \"kubernetes.io/projected/eece1c74-3904-47a7-ab7f-3464ffd6767e-kube-api-access-frzth\") pod \"payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd\" (UID: \"eece1c74-3904-47a7-ab7f-3464ffd6767e\") " pod="openshift-ingress/payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd" Oct 30 22:26:04.359289 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:04.359224 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd" Oct 30 22:26:04.744431 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:26:04.744389 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeece1c74_3904_47a7_ab7f_3464ffd6767e.slice/crio-d3aa89ed44da3497c944057fa153abf9bb536eee5aa58b3758512569bb4fad19 WatchSource:0}: Error finding container d3aa89ed44da3497c944057fa153abf9bb536eee5aa58b3758512569bb4fad19: Status 404 returned error can't find the container with id d3aa89ed44da3497c944057fa153abf9bb536eee5aa58b3758512569bb4fad19 Oct 30 22:26:04.748106 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:04.748037 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd"] Oct 30 22:26:05.090720 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:05.090680 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd" event={"ID":"eece1c74-3904-47a7-ab7f-3464ffd6767e","Type":"ContainerStarted","Data":"5ca86708cf567144143f532781ac56dad0377d42873692a1322867a47342bcbd"} Oct 30 22:26:05.090909 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:05.090740 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd" event={"ID":"eece1c74-3904-47a7-ab7f-3464ffd6767e","Type":"ContainerStarted","Data":"d3aa89ed44da3497c944057fa153abf9bb536eee5aa58b3758512569bb4fad19"} Oct 30 22:26:05.090909 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:05.090769 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd" Oct 30 22:26:05.211561 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:05.211503 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd" podStartSLOduration=2.211484497 podStartE2EDuration="2.211484497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:05.209392944 +0000 UTC m=+2167.761569243" watchObservedRunningTime="2026-10-30 22:26:05.211484497 +0000 UTC m=+2167.763660842" Oct 30 22:26:16.101293 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:16.101191 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd" Oct 30 22:26:23.281736 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:23.281688 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:26:27.916677 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:26:27.916631 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:26:28.984270 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:28.984224 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5"] Oct 30 22:26:28.989471 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:28.989443 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.043284 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.043214 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5637a7-openshift-default-dockercfg-m5lvg\"" Oct 30 22:26:29.047512 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.047480 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5"] Oct 30 22:26:29.088334 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.088291 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-workload-certs\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.088508 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.088450 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvs49\" (UniqueName: \"kubernetes.io/projected/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-kube-api-access-kvs49\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.088623 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.088605 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-podinfo\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.088688 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.088673 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-workload-socket\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.088795 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.088772 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-credential-socket\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.088962 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.088935 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-token\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.089051 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.089033 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istiod-ca-cert\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.089094 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.089083 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-envoy\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.089201 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.089182 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-data\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.190660 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.190607 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-token\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.190871 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.190697 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istiod-ca-cert\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.190871 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.190744 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-envoy\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.190871 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.190808 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-data\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.191065 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.190892 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-workload-certs\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.191065 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.190949 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvs49\" (UniqueName: \"kubernetes.io/projected/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-kube-api-access-kvs49\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.191176 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.191082 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-podinfo\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.191176 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.191133 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-workload-socket\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.191310 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.191183 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-credential-socket\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.191606 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.191579 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-credential-socket\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.191699 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.191636 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-data\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.191861 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.191835 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istiod-ca-cert\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.191943 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.191909 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-workload-certs\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.192151 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.192127 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-workload-socket\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.194229 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.194206 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-envoy\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.194388 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.194313 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-podinfo\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.254807 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.254730 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-token\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.256724 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.256697 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvs49\" (UniqueName: \"kubernetes.io/projected/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-kube-api-access-kvs49\") pod \"e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.304701 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.304661 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:29.773105 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:26:29.773073 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2760ed1_3b89_43bc_9c99_5748bb75fbc4.slice/crio-7fd594b448083d1b5604d943b28f44cf4851527cc6dcc1dbccd998a61b6bfa5e WatchSource:0}: Error finding container 7fd594b448083d1b5604d943b28f44cf4851527cc6dcc1dbccd998a61b6bfa5e: Status 404 returned error can't find the container with id 7fd594b448083d1b5604d943b28f44cf4851527cc6dcc1dbccd998a61b6bfa5e Oct 30 22:26:29.776555 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.776525 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5"] Oct 30 22:26:29.776684 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.776522 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:26:29.776684 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.776631 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:26:29.776684 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:29.776665 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:26:30.290797 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:30.290758 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" event={"ID":"d2760ed1-3b89-43bc-9c99-5748bb75fbc4","Type":"ContainerStarted","Data":"53f6186bf9841e4fcdc3943d46575acfe88b954475613bf41508723b3629b40e"} Oct 30 22:26:30.291335 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:30.290813 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" event={"ID":"d2760ed1-3b89-43bc-9c99-5748bb75fbc4","Type":"ContainerStarted","Data":"7fd594b448083d1b5604d943b28f44cf4851527cc6dcc1dbccd998a61b6bfa5e"} Oct 30 22:26:30.305107 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:30.305067 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:30.306868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:30.306835 6006 patch_prober.go:29] interesting pod/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 22:26:30.307027 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:30.306887 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" podUID="d2760ed1-3b89-43bc-9c99-5748bb75fbc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 22:26:30.518273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:30.512960 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" podStartSLOduration=2.512937696 podStartE2EDuration="2.512937696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:30.471391971 +0000 UTC m=+2193.023568272" watchObservedRunningTime="2026-10-30 22:26:30.512937696 +0000 UTC m=+2193.065113997" Oct 30 22:26:31.034355 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:31.034309 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6"] Oct 30 22:26:31.038861 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:31.038838 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6" Oct 30 22:26:31.092437 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:31.092398 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:26:31.093526 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:31.093496 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6"] Oct 30 22:26:31.139778 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:31.139737 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:26:31.141650 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:31.141608 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f37fb44-2790-4b49-b217-64873b88039b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6\" (UID: \"9f37fb44-2790-4b49-b217-64873b88039b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6" Oct 30 22:26:31.141824 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:31.141675 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvs88\" (UniqueName: \"kubernetes.io/projected/9f37fb44-2790-4b49-b217-64873b88039b-kube-api-access-bvs88\") pod \"maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6\" (UID: \"9f37fb44-2790-4b49-b217-64873b88039b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6" Oct 30 22:26:31.141905 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:31.141873 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f37fb44-2790-4b49-b217-64873b88039b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6\" (UID: \"9f37fb44-2790-4b49-b217-64873b88039b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6" Oct 30 22:26:31.243208 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:31.243166 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f37fb44-2790-4b49-b217-64873b88039b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6\" (UID: \"9f37fb44-2790-4b49-b217-64873b88039b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6" Oct 30 22:26:31.243419 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:31.243221 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvs88\" (UniqueName: \"kubernetes.io/projected/9f37fb44-2790-4b49-b217-64873b88039b-kube-api-access-bvs88\") pod \"maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6\" (UID: \"9f37fb44-2790-4b49-b217-64873b88039b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6" Oct 30 22:26:31.243419 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:31.243386 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f37fb44-2790-4b49-b217-64873b88039b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6\" (UID: \"9f37fb44-2790-4b49-b217-64873b88039b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6" Oct 30 22:26:31.244196 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:31.244163 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f37fb44-2790-4b49-b217-64873b88039b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6\" (UID: \"9f37fb44-2790-4b49-b217-64873b88039b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6" Oct 30 22:26:31.245789 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:31.245764 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f37fb44-2790-4b49-b217-64873b88039b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6\" (UID: \"9f37fb44-2790-4b49-b217-64873b88039b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6" Oct 30 22:26:31.335297 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:31.329836 6006 patch_prober.go:29] interesting pod/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 22:26:31.335297 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:31.329894 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" podUID="d2760ed1-3b89-43bc-9c99-5748bb75fbc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 22:26:31.335297 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:31.333118 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvs88\" (UniqueName: \"kubernetes.io/projected/9f37fb44-2790-4b49-b217-64873b88039b-kube-api-access-bvs88\") pod \"maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6\" (UID: \"9f37fb44-2790-4b49-b217-64873b88039b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6" Oct 30 22:26:31.355057 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:31.355019 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6" Oct 30 22:26:31.762013 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:26:31.761968 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f37fb44_2790_4b49_b217_64873b88039b.slice/crio-492f610b3e5ae7bd1bd8c85a1e2f95b98eb7909bac56b20600a30c13bd7bbac4 WatchSource:0}: Error finding container 492f610b3e5ae7bd1bd8c85a1e2f95b98eb7909bac56b20600a30c13bd7bbac4: Status 404 returned error can't find the container with id 492f610b3e5ae7bd1bd8c85a1e2f95b98eb7909bac56b20600a30c13bd7bbac4 Oct 30 22:26:31.765990 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:31.765962 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6"] Oct 30 22:26:32.325452 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:32.325408 6006 patch_prober.go:29] interesting pod/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 22:26:32.325684 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:32.325483 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" podUID="d2760ed1-3b89-43bc-9c99-5748bb75fbc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 22:26:32.333295 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:32.333244 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6" event={"ID":"9f37fb44-2790-4b49-b217-64873b88039b","Type":"ContainerStarted","Data":"9176da8d45fa1a7ef30ae42398a50c6af8765fd1a6a29b4e1266ba63d6a8d454"} Oct 30 22:26:32.333470 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:32.333317 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6" event={"ID":"9f37fb44-2790-4b49-b217-64873b88039b","Type":"ContainerStarted","Data":"492f610b3e5ae7bd1bd8c85a1e2f95b98eb7909bac56b20600a30c13bd7bbac4"} Oct 30 22:26:32.453910 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:32.453798 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6" podStartSLOduration=2.453781706 podStartE2EDuration="2.453781706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:32.451174861 +0000 UTC m=+2195.003351161" watchObservedRunningTime="2026-10-30 22:26:32.453781706 +0000 UTC m=+2195.005958011" Oct 30 22:26:33.323478 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:33.323438 6006 patch_prober.go:29] interesting pod/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 22:26:33.323709 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:33.323541 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" podUID="d2760ed1-3b89-43bc-9c99-5748bb75fbc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 22:26:33.343389 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:33.343350 6006 generic.go:356] "Generic (PLEG): container finished" podID="9f37fb44-2790-4b49-b217-64873b88039b" containerID="9176da8d45fa1a7ef30ae42398a50c6af8765fd1a6a29b4e1266ba63d6a8d454" exitCode=0 Oct 30 22:26:33.343588 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:33.343436 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6" event={"ID":"9f37fb44-2790-4b49-b217-64873b88039b","Type":"ContainerDied","Data":"9176da8d45fa1a7ef30ae42398a50c6af8765fd1a6a29b4e1266ba63d6a8d454"} Oct 30 22:26:34.312828 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:34.312779 6006 patch_prober.go:29] interesting pod/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 22:26:34.313515 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:34.313471 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" podUID="d2760ed1-3b89-43bc-9c99-5748bb75fbc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 22:26:34.675837 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:34.675803 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6" Oct 30 22:26:34.842766 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:34.841509 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f37fb44-2790-4b49-b217-64873b88039b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f37fb44-2790-4b49-b217-64873b88039b-maas-api-openshift-ca\") pod \"9f37fb44-2790-4b49-b217-64873b88039b\" (UID: \"9f37fb44-2790-4b49-b217-64873b88039b\") " Oct 30 22:26:34.842766 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:34.841630 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f37fb44-2790-4b49-b217-64873b88039b-kube-api-access-bvs88\" (UniqueName: \"kubernetes.io/projected/9f37fb44-2790-4b49-b217-64873b88039b-kube-api-access-bvs88\") pod \"9f37fb44-2790-4b49-b217-64873b88039b\" (UID: \"9f37fb44-2790-4b49-b217-64873b88039b\") " Oct 30 22:26:34.842766 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:34.841699 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f37fb44-2790-4b49-b217-64873b88039b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f37fb44-2790-4b49-b217-64873b88039b-maas-api-xks-ca\") pod \"9f37fb44-2790-4b49-b217-64873b88039b\" (UID: \"9f37fb44-2790-4b49-b217-64873b88039b\") " Oct 30 22:26:34.842766 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:34.842520 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f37fb44-2790-4b49-b217-64873b88039b-maas-api-openshift-ca" pod "9f37fb44-2790-4b49-b217-64873b88039b" (UID: "9f37fb44-2790-4b49-b217-64873b88039b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:34.851947 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:34.851905 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f37fb44-2790-4b49-b217-64873b88039b-kube-api-access-bvs88" pod "9f37fb44-2790-4b49-b217-64873b88039b" (UID: "9f37fb44-2790-4b49-b217-64873b88039b"). InnerVolumeSpecName "kube-api-access-bvs88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:34.854118 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:34.854080 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f37fb44-2790-4b49-b217-64873b88039b-maas-api-xks-ca" pod "9f37fb44-2790-4b49-b217-64873b88039b" (UID: "9f37fb44-2790-4b49-b217-64873b88039b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:34.943460 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:34.943357 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f37fb44-2790-4b49-b217-64873b88039b-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:34.943460 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:34.943396 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvs88\" (UniqueName: \"kubernetes.io/projected/9f37fb44-2790-4b49-b217-64873b88039b-kube-api-access-bvs88\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:34.943460 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:34.943412 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f37fb44-2790-4b49-b217-64873b88039b-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:35.307767 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:35.307722 6006 patch_prober.go:29] interesting pod/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 22:26:35.308033 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:35.308004 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" podUID="d2760ed1-3b89-43bc-9c99-5748bb75fbc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 22:26:35.362837 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:35.362808 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6" Oct 30 22:26:35.363234 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:35.362839 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6" event={"ID":"9f37fb44-2790-4b49-b217-64873b88039b","Type":"ContainerDied","Data":"492f610b3e5ae7bd1bd8c85a1e2f95b98eb7909bac56b20600a30c13bd7bbac4"} Oct 30 22:26:35.363234 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:35.362885 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="492f610b3e5ae7bd1bd8c85a1e2f95b98eb7909bac56b20600a30c13bd7bbac4" Oct 30 22:26:36.310217 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:36.310183 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:36.310640 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:36.310619 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:36.311244 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:26:36.311223 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:26:57.913006 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:26:57.912960 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:27:15.728332 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:15.728293 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:27:17.247499 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:17.247456 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp"] Oct 30 22:27:17.248165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:17.248148 6006 cpu_manager.go:414] "containerMap: removing container" podUID="9f37fb44-2790-4b49-b217-64873b88039b" containerName="revoke-keys" Oct 30 22:27:17.248227 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:17.248166 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f37fb44-2790-4b49-b217-64873b88039b" containerName="revoke-keys" Oct 30 22:27:17.248298 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:17.248287 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f37fb44-2790-4b49-b217-64873b88039b" containerName="revoke-keys" Oct 30 22:27:17.253238 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:17.253217 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp" Oct 30 22:27:17.349406 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:17.349365 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:27:17.349619 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:17.349505 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:27:17.355043 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:17.354998 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp"] Oct 30 22:27:17.367531 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:17.367502 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63ed983f-f4cb-4df0-ad0e-b7df9364babe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp\" (UID: \"63ed983f-f4cb-4df0-ad0e-b7df9364babe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp" Oct 30 22:27:17.367681 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:17.367627 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpgwp\" (UniqueName: \"kubernetes.io/projected/63ed983f-f4cb-4df0-ad0e-b7df9364babe-kube-api-access-cpgwp\") pod \"maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp\" (UID: \"63ed983f-f4cb-4df0-ad0e-b7df9364babe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp" Oct 30 22:27:17.367785 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:17.367769 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63ed983f-f4cb-4df0-ad0e-b7df9364babe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp\" (UID: \"63ed983f-f4cb-4df0-ad0e-b7df9364babe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp" Oct 30 22:27:17.468765 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:17.468719 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpgwp\" (UniqueName: \"kubernetes.io/projected/63ed983f-f4cb-4df0-ad0e-b7df9364babe-kube-api-access-cpgwp\") pod \"maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp\" (UID: \"63ed983f-f4cb-4df0-ad0e-b7df9364babe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp" Oct 30 22:27:17.468955 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:17.468854 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63ed983f-f4cb-4df0-ad0e-b7df9364babe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp\" (UID: \"63ed983f-f4cb-4df0-ad0e-b7df9364babe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp" Oct 30 22:27:17.469005 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:17.468972 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63ed983f-f4cb-4df0-ad0e-b7df9364babe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp\" (UID: \"63ed983f-f4cb-4df0-ad0e-b7df9364babe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp" Oct 30 22:27:17.469616 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:17.469587 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63ed983f-f4cb-4df0-ad0e-b7df9364babe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp\" (UID: \"63ed983f-f4cb-4df0-ad0e-b7df9364babe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp" Oct 30 22:27:17.471469 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:17.471450 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63ed983f-f4cb-4df0-ad0e-b7df9364babe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp\" (UID: \"63ed983f-f4cb-4df0-ad0e-b7df9364babe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp" Oct 30 22:27:17.527149 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:17.527075 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpgwp\" (UniqueName: \"kubernetes.io/projected/63ed983f-f4cb-4df0-ad0e-b7df9364babe-kube-api-access-cpgwp\") pod \"maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp\" (UID: \"63ed983f-f4cb-4df0-ad0e-b7df9364babe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp" Oct 30 22:27:17.568279 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:17.568229 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp" Oct 30 22:27:17.943050 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:17.943018 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp"] Oct 30 22:27:17.944409 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:27:17.944383 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ed983f_f4cb_4df0_ad0e_b7df9364babe.slice/crio-652a79b6d8f6686989f1faaca805b149f95ae09c19e43351f276d460df17bb71 WatchSource:0}: Error finding container 652a79b6d8f6686989f1faaca805b149f95ae09c19e43351f276d460df17bb71: Status 404 returned error can't find the container with id 652a79b6d8f6686989f1faaca805b149f95ae09c19e43351f276d460df17bb71 Oct 30 22:27:18.730240 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:18.730203 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp" event={"ID":"63ed983f-f4cb-4df0-ad0e-b7df9364babe","Type":"ContainerStarted","Data":"ab74db2b9aa52ec51cd0e1e4a4fac42e03663f8a7ae36019e35ec0f9e8abf565"} Oct 30 22:27:18.730240 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:18.730276 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp" event={"ID":"63ed983f-f4cb-4df0-ad0e-b7df9364babe","Type":"ContainerStarted","Data":"652a79b6d8f6686989f1faaca805b149f95ae09c19e43351f276d460df17bb71"} Oct 30 22:27:18.888097 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:18.888036 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp" podStartSLOduration=1.888021626 podStartE2EDuration="1.888021626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:18.83998144 +0000 UTC m=+2241.392157741" watchObservedRunningTime="2026-10-30 22:27:18.888021626 +0000 UTC m=+2241.440197925" Oct 30 22:27:19.738627 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:19.738588 6006 generic.go:356] "Generic (PLEG): container finished" podID="63ed983f-f4cb-4df0-ad0e-b7df9364babe" containerID="ab74db2b9aa52ec51cd0e1e4a4fac42e03663f8a7ae36019e35ec0f9e8abf565" exitCode=0 Oct 30 22:27:19.739039 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:19.738673 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp" event={"ID":"63ed983f-f4cb-4df0-ad0e-b7df9364babe","Type":"ContainerDied","Data":"ab74db2b9aa52ec51cd0e1e4a4fac42e03663f8a7ae36019e35ec0f9e8abf565"} Oct 30 22:27:20.930351 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:20.930321 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp" Oct 30 22:27:21.024043 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:21.023943 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63ed983f-f4cb-4df0-ad0e-b7df9364babe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63ed983f-f4cb-4df0-ad0e-b7df9364babe-maas-api-xks-ca\") pod \"63ed983f-f4cb-4df0-ad0e-b7df9364babe\" (UID: \"63ed983f-f4cb-4df0-ad0e-b7df9364babe\") " Oct 30 22:27:21.024216 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:21.024098 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63ed983f-f4cb-4df0-ad0e-b7df9364babe-kube-api-access-cpgwp\" (UniqueName: \"kubernetes.io/projected/63ed983f-f4cb-4df0-ad0e-b7df9364babe-kube-api-access-cpgwp\") pod \"63ed983f-f4cb-4df0-ad0e-b7df9364babe\" (UID: \"63ed983f-f4cb-4df0-ad0e-b7df9364babe\") " Oct 30 22:27:21.024216 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:21.024139 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63ed983f-f4cb-4df0-ad0e-b7df9364babe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63ed983f-f4cb-4df0-ad0e-b7df9364babe-maas-api-openshift-ca\") pod \"63ed983f-f4cb-4df0-ad0e-b7df9364babe\" (UID: \"63ed983f-f4cb-4df0-ad0e-b7df9364babe\") " Oct 30 22:27:21.024619 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:21.024591 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63ed983f-f4cb-4df0-ad0e-b7df9364babe-maas-api-openshift-ca" pod "63ed983f-f4cb-4df0-ad0e-b7df9364babe" (UID: "63ed983f-f4cb-4df0-ad0e-b7df9364babe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:21.024803 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:21.024785 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63ed983f-f4cb-4df0-ad0e-b7df9364babe-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:21.026297 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:21.026264 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63ed983f-f4cb-4df0-ad0e-b7df9364babe-maas-api-xks-ca" pod "63ed983f-f4cb-4df0-ad0e-b7df9364babe" (UID: "63ed983f-f4cb-4df0-ad0e-b7df9364babe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:21.026550 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:21.026529 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ed983f-f4cb-4df0-ad0e-b7df9364babe-kube-api-access-cpgwp" pod "63ed983f-f4cb-4df0-ad0e-b7df9364babe" (UID: "63ed983f-f4cb-4df0-ad0e-b7df9364babe"). InnerVolumeSpecName "kube-api-access-cpgwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:21.126624 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:21.126585 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63ed983f-f4cb-4df0-ad0e-b7df9364babe-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:21.126624 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:21.126621 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpgwp\" (UniqueName: \"kubernetes.io/projected/63ed983f-f4cb-4df0-ad0e-b7df9364babe-kube-api-access-cpgwp\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:21.754029 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:21.753997 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp" Oct 30 22:27:21.754237 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:21.753988 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp" event={"ID":"63ed983f-f4cb-4df0-ad0e-b7df9364babe","Type":"ContainerDied","Data":"652a79b6d8f6686989f1faaca805b149f95ae09c19e43351f276d460df17bb71"} Oct 30 22:27:21.754237 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:21.754157 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="652a79b6d8f6686989f1faaca805b149f95ae09c19e43351f276d460df17bb71" Oct 30 22:27:22.224409 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:22.224370 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw"] Oct 30 22:27:22.225428 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:22.225405 6006 cpu_manager.go:414] "containerMap: removing container" podUID="63ed983f-f4cb-4df0-ad0e-b7df9364babe" containerName="revoke-keys" Oct 30 22:27:22.225496 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:22.225430 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ed983f-f4cb-4df0-ad0e-b7df9364babe" containerName="revoke-keys" Oct 30 22:27:22.225640 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:22.225628 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ed983f-f4cb-4df0-ad0e-b7df9364babe" containerName="revoke-keys" Oct 30 22:27:22.230079 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:22.230057 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw" Oct 30 22:27:22.280462 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:22.280418 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:27:22.282714 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:22.282683 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw"] Oct 30 22:27:22.324699 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:22.324663 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:27:22.342574 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:22.342538 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2762p\" (UniqueName: \"kubernetes.io/projected/51b0ff5c-2571-4224-9ed5-d40513e49ae5-kube-api-access-2762p\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw\" (UID: \"51b0ff5c-2571-4224-9ed5-d40513e49ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw" Oct 30 22:27:22.342757 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:22.342659 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b0ff5c-2571-4224-9ed5-d40513e49ae5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw\" (UID: \"51b0ff5c-2571-4224-9ed5-d40513e49ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw" Oct 30 22:27:22.342757 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:22.342705 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b0ff5c-2571-4224-9ed5-d40513e49ae5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw\" (UID: \"51b0ff5c-2571-4224-9ed5-d40513e49ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw" Oct 30 22:27:22.444532 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:22.444494 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b0ff5c-2571-4224-9ed5-d40513e49ae5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw\" (UID: \"51b0ff5c-2571-4224-9ed5-d40513e49ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw" Oct 30 22:27:22.444723 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:22.444550 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b0ff5c-2571-4224-9ed5-d40513e49ae5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw\" (UID: \"51b0ff5c-2571-4224-9ed5-d40513e49ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw" Oct 30 22:27:22.444771 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:22.444755 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2762p\" (UniqueName: \"kubernetes.io/projected/51b0ff5c-2571-4224-9ed5-d40513e49ae5-kube-api-access-2762p\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw\" (UID: \"51b0ff5c-2571-4224-9ed5-d40513e49ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw" Oct 30 22:27:22.445548 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:22.445511 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b0ff5c-2571-4224-9ed5-d40513e49ae5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw\" (UID: \"51b0ff5c-2571-4224-9ed5-d40513e49ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw" Oct 30 22:27:22.447074 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:22.447042 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b0ff5c-2571-4224-9ed5-d40513e49ae5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw\" (UID: \"51b0ff5c-2571-4224-9ed5-d40513e49ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw" Oct 30 22:27:22.503640 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:22.503562 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2762p\" (UniqueName: \"kubernetes.io/projected/51b0ff5c-2571-4224-9ed5-d40513e49ae5-kube-api-access-2762p\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw\" (UID: \"51b0ff5c-2571-4224-9ed5-d40513e49ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw" Oct 30 22:27:22.545983 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:22.545947 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw" Oct 30 22:27:22.921679 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:22.921644 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw"] Oct 30 22:27:22.922692 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:27:22.922659 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b0ff5c_2571_4224_9ed5_d40513e49ae5.slice/crio-8672847df0889a5d251c38fdd22845c4e39921992af363f0aac4929d422bc476 WatchSource:0}: Error finding container 8672847df0889a5d251c38fdd22845c4e39921992af363f0aac4929d422bc476: Status 404 returned error can't find the container with id 8672847df0889a5d251c38fdd22845c4e39921992af363f0aac4929d422bc476 Oct 30 22:27:23.311866 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:23.311825 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5"] Oct 30 22:27:23.312446 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:23.312079 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" podUID="500679e9-0b27-49c3-9b36-7fcfb7d1fc32" containerName="istio-proxy" containerID="cri-o://6d061c26c259af7af648cb0611c86554a942bdc9b7389bc149b075371098778e" gracePeriod=30 Oct 30 22:27:23.770070 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:23.770028 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw" event={"ID":"51b0ff5c-2571-4224-9ed5-d40513e49ae5","Type":"ContainerStarted","Data":"5a314bdc8e01fd3b160c15712f9453430c15279c2a526395baf5c2c3cc9e9a7a"} Oct 30 22:27:23.770285 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:23.770115 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw" event={"ID":"51b0ff5c-2571-4224-9ed5-d40513e49ae5","Type":"ContainerStarted","Data":"8672847df0889a5d251c38fdd22845c4e39921992af363f0aac4929d422bc476"} Oct 30 22:27:24.779667 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:24.779623 6006 generic.go:356] "Generic (PLEG): container finished" podID="51b0ff5c-2571-4224-9ed5-d40513e49ae5" containerID="5a314bdc8e01fd3b160c15712f9453430c15279c2a526395baf5c2c3cc9e9a7a" exitCode=0 Oct 30 22:27:24.780188 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:24.779706 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw" event={"ID":"51b0ff5c-2571-4224-9ed5-d40513e49ae5","Type":"ContainerDied","Data":"5a314bdc8e01fd3b160c15712f9453430c15279c2a526395baf5c2c3cc9e9a7a"} Oct 30 22:27:25.972855 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:25.972824 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw" Oct 30 22:27:26.098525 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:26.098482 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51b0ff5c-2571-4224-9ed5-d40513e49ae5-kube-api-access-2762p\" (UniqueName: \"kubernetes.io/projected/51b0ff5c-2571-4224-9ed5-d40513e49ae5-kube-api-access-2762p\") pod \"51b0ff5c-2571-4224-9ed5-d40513e49ae5\" (UID: \"51b0ff5c-2571-4224-9ed5-d40513e49ae5\") " Oct 30 22:27:26.098723 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:26.098652 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51b0ff5c-2571-4224-9ed5-d40513e49ae5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b0ff5c-2571-4224-9ed5-d40513e49ae5-maas-api-xks-ca\") pod \"51b0ff5c-2571-4224-9ed5-d40513e49ae5\" (UID: \"51b0ff5c-2571-4224-9ed5-d40513e49ae5\") " Oct 30 22:27:26.098723 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:26.098683 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51b0ff5c-2571-4224-9ed5-d40513e49ae5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b0ff5c-2571-4224-9ed5-d40513e49ae5-maas-api-openshift-ca\") pod \"51b0ff5c-2571-4224-9ed5-d40513e49ae5\" (UID: \"51b0ff5c-2571-4224-9ed5-d40513e49ae5\") " Oct 30 22:27:26.099118 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:26.099086 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b0ff5c-2571-4224-9ed5-d40513e49ae5-maas-api-openshift-ca" pod "51b0ff5c-2571-4224-9ed5-d40513e49ae5" (UID: "51b0ff5c-2571-4224-9ed5-d40513e49ae5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:26.099414 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:26.099390 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b0ff5c-2571-4224-9ed5-d40513e49ae5-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:26.101221 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:26.101185 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51b0ff5c-2571-4224-9ed5-d40513e49ae5-maas-api-xks-ca" pod "51b0ff5c-2571-4224-9ed5-d40513e49ae5" (UID: "51b0ff5c-2571-4224-9ed5-d40513e49ae5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:26.107464 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:26.107426 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b0ff5c-2571-4224-9ed5-d40513e49ae5-kube-api-access-2762p" pod "51b0ff5c-2571-4224-9ed5-d40513e49ae5" (UID: "51b0ff5c-2571-4224-9ed5-d40513e49ae5"). InnerVolumeSpecName "kube-api-access-2762p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:26.201338 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:26.201296 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2762p\" (UniqueName: \"kubernetes.io/projected/51b0ff5c-2571-4224-9ed5-d40513e49ae5-kube-api-access-2762p\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:26.201338 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:26.201336 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b0ff5c-2571-4224-9ed5-d40513e49ae5-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:26.497051 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:26.496961 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:27:26.497238 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:26.497046 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" podUID="500679e9-0b27-49c3-9b36-7fcfb7d1fc32" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:27:26.800274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:26.795630 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw" Oct 30 22:27:26.800274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:26.795660 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw" event={"ID":"51b0ff5c-2571-4224-9ed5-d40513e49ae5","Type":"ContainerDied","Data":"8672847df0889a5d251c38fdd22845c4e39921992af363f0aac4929d422bc476"} Oct 30 22:27:26.800274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:26.795730 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8672847df0889a5d251c38fdd22845c4e39921992af363f0aac4929d422bc476" Oct 30 22:27:27.913038 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:27:27.912997 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:27:28.612198 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.612164 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:27:28.738352 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.738314 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-data\") pod \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " Oct 30 22:27:28.739060 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.739040 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istiod-ca-cert\") pod \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " Oct 30 22:27:28.739195 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.738977 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-data" pod "500679e9-0b27-49c3-9b36-7fcfb7d1fc32" (UID: "500679e9-0b27-49c3-9b36-7fcfb7d1fc32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:28.739195 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.739146 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-workload-certs\") pod \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " Oct 30 22:27:28.739195 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.739176 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-workload-socket\") pod \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " Oct 30 22:27:28.739424 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.739281 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-token\" (UniqueName: \"kubernetes.io/projected/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-token\") pod \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " Oct 30 22:27:28.739424 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.739318 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-podinfo\") pod \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " Oct 30 22:27:28.739518 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.739480 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-workload-certs" pod "500679e9-0b27-49c3-9b36-7fcfb7d1fc32" (UID: "500679e9-0b27-49c3-9b36-7fcfb7d1fc32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:28.739577 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.739551 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-envoy\") pod \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " Oct 30 22:27:28.739635 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.739584 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-kube-api-access-b4qwc\" (UniqueName: \"kubernetes.io/projected/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-kube-api-access-b4qwc\") pod \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " Oct 30 22:27:28.739705 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.739596 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istiod-ca-cert" pod "500679e9-0b27-49c3-9b36-7fcfb7d1fc32" (UID: "500679e9-0b27-49c3-9b36-7fcfb7d1fc32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:28.739770 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.739678 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-workload-socket" pod "500679e9-0b27-49c3-9b36-7fcfb7d1fc32" (UID: "500679e9-0b27-49c3-9b36-7fcfb7d1fc32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:28.739770 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.739623 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-credential-socket\") pod \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\" (UID: \"500679e9-0b27-49c3-9b36-7fcfb7d1fc32\") " Oct 30 22:27:28.739880 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.739822 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-credential-socket" pod "500679e9-0b27-49c3-9b36-7fcfb7d1fc32" (UID: "500679e9-0b27-49c3-9b36-7fcfb7d1fc32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:28.740629 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.740601 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:28.740744 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.740634 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:28.740744 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.740649 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:28.740744 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.740665 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:28.740744 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.740679 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:28.742415 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.742394 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-envoy" pod "500679e9-0b27-49c3-9b36-7fcfb7d1fc32" (UID: "500679e9-0b27-49c3-9b36-7fcfb7d1fc32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:28.742767 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.742742 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-token" pod "500679e9-0b27-49c3-9b36-7fcfb7d1fc32" (UID: "500679e9-0b27-49c3-9b36-7fcfb7d1fc32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:28.743023 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.742999 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-podinfo" pod "500679e9-0b27-49c3-9b36-7fcfb7d1fc32" (UID: "500679e9-0b27-49c3-9b36-7fcfb7d1fc32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:27:28.744278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.744240 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-kube-api-access-b4qwc" pod "500679e9-0b27-49c3-9b36-7fcfb7d1fc32" (UID: "500679e9-0b27-49c3-9b36-7fcfb7d1fc32"). InnerVolumeSpecName "kube-api-access-b4qwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:28.816375 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.816337 6006 generic.go:356] "Generic (PLEG): container finished" podID="500679e9-0b27-49c3-9b36-7fcfb7d1fc32" containerID="6d061c26c259af7af648cb0611c86554a942bdc9b7389bc149b075371098778e" exitCode=0 Oct 30 22:27:28.816552 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.816414 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" Oct 30 22:27:28.816552 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.816434 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" event={"ID":"500679e9-0b27-49c3-9b36-7fcfb7d1fc32","Type":"ContainerDied","Data":"6d061c26c259af7af648cb0611c86554a942bdc9b7389bc149b075371098778e"} Oct 30 22:27:28.816552 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.816503 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5" event={"ID":"500679e9-0b27-49c3-9b36-7fcfb7d1fc32","Type":"ContainerDied","Data":"9abb7ab6bf87281395b38f305530e0c66c4bb2b70e33bcb7f8080139999aad49"} Oct 30 22:27:28.816552 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.816533 6006 scope.go:122] "RemoveContainer" containerID="6d061c26c259af7af648cb0611c86554a942bdc9b7389bc149b075371098778e" Oct 30 22:27:28.835344 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.835317 6006 scope.go:122] "RemoveContainer" containerID="6d061c26c259af7af648cb0611c86554a942bdc9b7389bc149b075371098778e" Oct 30 22:27:28.835689 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:27:28.835651 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d061c26c259af7af648cb0611c86554a942bdc9b7389bc149b075371098778e\": container with ID starting with 6d061c26c259af7af648cb0611c86554a942bdc9b7389bc149b075371098778e not found: ID does not exist" containerID="6d061c26c259af7af648cb0611c86554a942bdc9b7389bc149b075371098778e" Oct 30 22:27:28.835762 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.835689 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d061c26c259af7af648cb0611c86554a942bdc9b7389bc149b075371098778e"} err="failed to get container status \"6d061c26c259af7af648cb0611c86554a942bdc9b7389bc149b075371098778e\": rpc error: code = NotFound desc = could not find container \"6d061c26c259af7af648cb0611c86554a942bdc9b7389bc149b075371098778e\": container with ID starting with 6d061c26c259af7af648cb0611c86554a942bdc9b7389bc149b075371098778e not found: ID does not exist" Oct 30 22:27:28.843461 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.843383 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:28.843461 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.843417 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:28.843461 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.843435 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:28.843461 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.843451 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4qwc\" (UniqueName: \"kubernetes.io/projected/500679e9-0b27-49c3-9b36-7fcfb7d1fc32-kube-api-access-b4qwc\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:28.979552 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:28.979518 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5"] Oct 30 22:27:29.024423 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:29.024385 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-qktm5"] Oct 30 22:27:29.607583 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:29.607545 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="500679e9-0b27-49c3-9b36-7fcfb7d1fc32" path="/var/lib/kubelet/pods/500679e9-0b27-49c3-9b36-7fcfb7d1fc32/volumes" Oct 30 22:27:29.685124 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:29.685087 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw"] Oct 30 22:27:29.685342 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:29.685295 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" podUID="d255a3e1-7f5a-40a5-98e5-3d481b133f0e" containerName="maas-api" containerID="cri-o://b3fe7ef4c21797e146bbc9c85d93476878d58bf6b6e30f190ddfcb8ba7003da3" gracePeriod=30 Oct 30 22:27:29.831094 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:29.831040 6006 generic.go:356] "Generic (PLEG): container finished" podID="d255a3e1-7f5a-40a5-98e5-3d481b133f0e" containerID="b3fe7ef4c21797e146bbc9c85d93476878d58bf6b6e30f190ddfcb8ba7003da3" exitCode=0 Oct 30 22:27:29.831331 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:29.831208 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" event={"ID":"d255a3e1-7f5a-40a5-98e5-3d481b133f0e","Type":"ContainerDied","Data":"b3fe7ef4c21797e146bbc9c85d93476878d58bf6b6e30f190ddfcb8ba7003da3"} Oct 30 22:27:30.398914 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:30.398877 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" Oct 30 22:27:30.480259 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:30.480204 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-metrics-certs\") pod \"d255a3e1-7f5a-40a5-98e5-3d481b133f0e\" (UID: \"d255a3e1-7f5a-40a5-98e5-3d481b133f0e\") " Oct 30 22:27:30.480461 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:30.480325 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-kube-api-access-lcltk\" (UniqueName: \"kubernetes.io/projected/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-kube-api-access-lcltk\") pod \"d255a3e1-7f5a-40a5-98e5-3d481b133f0e\" (UID: \"d255a3e1-7f5a-40a5-98e5-3d481b133f0e\") " Oct 30 22:27:30.480538 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:30.480511 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-maas-api-tls\") pod \"d255a3e1-7f5a-40a5-98e5-3d481b133f0e\" (UID: \"d255a3e1-7f5a-40a5-98e5-3d481b133f0e\") " Oct 30 22:27:30.480601 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:30.480559 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-openshift-service-ca\") pod \"d255a3e1-7f5a-40a5-98e5-3d481b133f0e\" (UID: \"d255a3e1-7f5a-40a5-98e5-3d481b133f0e\") " Oct 30 22:27:30.480908 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:30.480875 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-openshift-service-ca" pod "d255a3e1-7f5a-40a5-98e5-3d481b133f0e" (UID: "d255a3e1-7f5a-40a5-98e5-3d481b133f0e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:30.481303 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:30.481279 6006 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-openshift-service-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:30.482678 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:30.482652 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-maas-api-tls" pod "d255a3e1-7f5a-40a5-98e5-3d481b133f0e" (UID: "d255a3e1-7f5a-40a5-98e5-3d481b133f0e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:30.482678 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:30.482670 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-metrics-certs" pod "d255a3e1-7f5a-40a5-98e5-3d481b133f0e" (UID: "d255a3e1-7f5a-40a5-98e5-3d481b133f0e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:30.482808 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:30.482668 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-kube-api-access-lcltk" pod "d255a3e1-7f5a-40a5-98e5-3d481b133f0e" (UID: "d255a3e1-7f5a-40a5-98e5-3d481b133f0e"). InnerVolumeSpecName "kube-api-access-lcltk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:30.583065 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:30.583023 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-maas-api-tls\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:30.583065 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:30.583063 6006 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-metrics-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:30.583321 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:30.583080 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcltk\" (UniqueName: \"kubernetes.io/projected/d255a3e1-7f5a-40a5-98e5-3d481b133f0e-kube-api-access-lcltk\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:30.845432 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:30.845403 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" Oct 30 22:27:30.845643 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:30.845444 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" event={"ID":"d255a3e1-7f5a-40a5-98e5-3d481b133f0e","Type":"ContainerDied","Data":"35d5dc11d15fb05fc1e170e966a0119b749b161263b373147fd8b4ec6cfe13bb"} Oct 30 22:27:30.845643 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:30.845521 6006 scope.go:122] "RemoveContainer" containerID="b3fe7ef4c21797e146bbc9c85d93476878d58bf6b6e30f190ddfcb8ba7003da3" Oct 30 22:27:31.003534 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:31.003494 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw"] Oct 30 22:27:31.051241 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:31.051201 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw"] Oct 30 22:27:31.607081 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:31.607031 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d255a3e1-7f5a-40a5-98e5-3d481b133f0e" path="/var/lib/kubelet/pods/d255a3e1-7f5a-40a5-98e5-3d481b133f0e/volumes" Oct 30 22:27:33.040800 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:33.040741 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5637a7-f49dc666f-56bdw" podUID="d255a3e1-7f5a-40a5-98e5-3d481b133f0e" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.137:8443/health\": context deadline exceeded" Oct 30 22:27:33.606398 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:33.606360 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp"] Oct 30 22:27:33.647360 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:33.647322 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5637a7-ts2dp"] Oct 30 22:27:35.605413 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:35.605375 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63ed983f-f4cb-4df0-ad0e-b7df9364babe" path="/var/lib/kubelet/pods/63ed983f-f4cb-4df0-ad0e-b7df9364babe/volumes" Oct 30 22:27:40.773659 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:40.773618 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5"] Oct 30 22:27:40.774070 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:40.773894 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" podUID="d2760ed1-3b89-43bc-9c99-5748bb75fbc4" containerName="istio-proxy" containerID="cri-o://53f6186bf9841e4fcdc3943d46575acfe88b954475613bf41508723b3629b40e" gracePeriod=30 Oct 30 22:27:43.213289 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:43.209492 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk"] Oct 30 22:27:43.213289 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:43.211197 6006 cpu_manager.go:414] "containerMap: removing container" podUID="500679e9-0b27-49c3-9b36-7fcfb7d1fc32" containerName="istio-proxy" Oct 30 22:27:43.213289 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:43.211223 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="500679e9-0b27-49c3-9b36-7fcfb7d1fc32" containerName="istio-proxy" Oct 30 22:27:43.213289 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:43.211263 6006 cpu_manager.go:414] "containerMap: removing container" podUID="d255a3e1-7f5a-40a5-98e5-3d481b133f0e" containerName="maas-api" Oct 30 22:27:43.213289 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:43.211271 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d255a3e1-7f5a-40a5-98e5-3d481b133f0e" containerName="maas-api" Oct 30 22:27:43.213289 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:43.211283 6006 cpu_manager.go:414] "containerMap: removing container" podUID="51b0ff5c-2571-4224-9ed5-d40513e49ae5" containerName="revoke-keys" Oct 30 22:27:43.213289 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:43.211292 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51b0ff5c-2571-4224-9ed5-d40513e49ae5" containerName="revoke-keys" Oct 30 22:27:43.213289 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:43.211922 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="500679e9-0b27-49c3-9b36-7fcfb7d1fc32" containerName="istio-proxy" Oct 30 22:27:43.213289 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:43.211949 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="d255a3e1-7f5a-40a5-98e5-3d481b133f0e" containerName="maas-api" Oct 30 22:27:43.213289 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:43.211957 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="51b0ff5c-2571-4224-9ed5-d40513e49ae5" containerName="revoke-keys" Oct 30 22:27:43.231181 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:43.231144 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk" Oct 30 22:27:43.269994 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:43.269951 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk"] Oct 30 22:27:43.323716 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:43.323667 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:27:43.350308 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:43.348841 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:27:43.398282 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:43.391037 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cckzx\" (UniqueName: \"kubernetes.io/projected/8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc-kube-api-access-cckzx\") pod \"maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk\" (UID: \"8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk" Oct 30 22:27:43.398728 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:43.398699 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk\" (UID: \"8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk" Oct 30 22:27:43.406539 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:43.406501 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk\" (UID: \"8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk" Oct 30 22:27:43.508342 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:43.508213 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cckzx\" (UniqueName: \"kubernetes.io/projected/8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc-kube-api-access-cckzx\") pod \"maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk\" (UID: \"8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk" Oct 30 22:27:43.508533 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:43.508408 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk\" (UID: \"8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk" Oct 30 22:27:43.508533 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:43.508521 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk\" (UID: \"8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk" Oct 30 22:27:43.509176 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:43.509149 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk\" (UID: \"8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk" Oct 30 22:27:43.510896 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:43.510866 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk\" (UID: \"8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk" Oct 30 22:27:43.569886 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:43.569845 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cckzx\" (UniqueName: \"kubernetes.io/projected/8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc-kube-api-access-cckzx\") pod \"maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk\" (UID: \"8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk" Oct 30 22:27:43.848340 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:43.848308 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk" Oct 30 22:27:44.365549 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:44.365500 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk"] Oct 30 22:27:44.465194 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:44.465144 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6"] Oct 30 22:27:44.485451 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:44.485421 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6" Oct 30 22:27:44.569325 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:44.567815 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6"] Oct 30 22:27:44.653465 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:44.653349 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0667a5-c67a-46fd-8728-1fc6a950eb58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6\" (UID: \"4a0667a5-c67a-46fd-8728-1fc6a950eb58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6" Oct 30 22:27:44.653465 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:44.653427 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xqcf\" (UniqueName: \"kubernetes.io/projected/4a0667a5-c67a-46fd-8728-1fc6a950eb58-kube-api-access-8xqcf\") pod \"maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6\" (UID: \"4a0667a5-c67a-46fd-8728-1fc6a950eb58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6" Oct 30 22:27:44.653739 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:44.653570 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0667a5-c67a-46fd-8728-1fc6a950eb58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6\" (UID: \"4a0667a5-c67a-46fd-8728-1fc6a950eb58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6" Oct 30 22:27:44.754599 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:44.754565 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0667a5-c67a-46fd-8728-1fc6a950eb58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6\" (UID: \"4a0667a5-c67a-46fd-8728-1fc6a950eb58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6" Oct 30 22:27:44.754815 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:44.754624 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xqcf\" (UniqueName: \"kubernetes.io/projected/4a0667a5-c67a-46fd-8728-1fc6a950eb58-kube-api-access-8xqcf\") pod \"maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6\" (UID: \"4a0667a5-c67a-46fd-8728-1fc6a950eb58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6" Oct 30 22:27:44.754815 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:44.754765 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0667a5-c67a-46fd-8728-1fc6a950eb58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6\" (UID: \"4a0667a5-c67a-46fd-8728-1fc6a950eb58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6" Oct 30 22:27:44.755410 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:44.755388 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0667a5-c67a-46fd-8728-1fc6a950eb58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6\" (UID: \"4a0667a5-c67a-46fd-8728-1fc6a950eb58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6" Oct 30 22:27:44.760700 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:44.757844 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0667a5-c67a-46fd-8728-1fc6a950eb58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6\" (UID: \"4a0667a5-c67a-46fd-8728-1fc6a950eb58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6" Oct 30 22:27:44.853312 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:44.852163 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xqcf\" (UniqueName: \"kubernetes.io/projected/4a0667a5-c67a-46fd-8728-1fc6a950eb58-kube-api-access-8xqcf\") pod \"maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6\" (UID: \"4a0667a5-c67a-46fd-8728-1fc6a950eb58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6" Oct 30 22:27:45.131277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:45.122700 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6" Oct 30 22:27:45.179965 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:45.179905 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk" event={"ID":"8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc","Type":"ContainerStarted","Data":"bb4705bb52c8a7fd275dd3a84552fa8d54ff4590d1bda5fccf4db60abe8e1ff1"} Oct 30 22:27:45.180138 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:45.180002 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk" event={"ID":"8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc","Type":"ContainerStarted","Data":"06fad2fef9d13d6dec93b942c94fd789eb58c26d4912756254e71e3d23ab2a54"} Oct 30 22:27:45.297781 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:45.297728 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk" podStartSLOduration=2.297713534 podStartE2EDuration="2.297713534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:45.294786854 +0000 UTC m=+2267.846963154" watchObservedRunningTime="2026-10-30 22:27:45.297713534 +0000 UTC m=+2267.849889900" Oct 30 22:27:45.583625 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:45.583595 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6"] Oct 30 22:27:45.585275 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:27:45.585222 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0667a5_c67a_46fd_8728_1fc6a950eb58.slice/crio-c209af8659bb7b0a5a8a4daf630c8ed8a8e4e52d4ebf5cd11b58bf9ac75253d7 WatchSource:0}: Error finding container c209af8659bb7b0a5a8a4daf630c8ed8a8e4e52d4ebf5cd11b58bf9ac75253d7: Status 404 returned error can't find the container with id c209af8659bb7b0a5a8a4daf630c8ed8a8e4e52d4ebf5cd11b58bf9ac75253d7 Oct 30 22:27:46.195742 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.195696 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6" event={"ID":"4a0667a5-c67a-46fd-8728-1fc6a950eb58","Type":"ContainerStarted","Data":"d8a42bf2cdd274afb0b5cc456248b26889927fce9c8123cef5b8ae850f7ecc52"} Oct 30 22:27:46.195898 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.195771 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6" event={"ID":"4a0667a5-c67a-46fd-8728-1fc6a950eb58","Type":"ContainerStarted","Data":"c209af8659bb7b0a5a8a4daf630c8ed8a8e4e52d4ebf5cd11b58bf9ac75253d7"} Oct 30 22:27:46.197485 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.197454 6006 generic.go:356] "Generic (PLEG): container finished" podID="8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc" containerID="bb4705bb52c8a7fd275dd3a84552fa8d54ff4590d1bda5fccf4db60abe8e1ff1" exitCode=0 Oct 30 22:27:46.197612 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.197533 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk" event={"ID":"8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc","Type":"ContainerDied","Data":"bb4705bb52c8a7fd275dd3a84552fa8d54ff4590d1bda5fccf4db60abe8e1ff1"} Oct 30 22:27:46.243277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.243184 6006 generic.go:356] "Generic (PLEG): container finished" podID="d2760ed1-3b89-43bc-9c99-5748bb75fbc4" containerID="53f6186bf9841e4fcdc3943d46575acfe88b954475613bf41508723b3629b40e" exitCode=0 Oct 30 22:27:46.243485 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.243429 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" event={"ID":"d2760ed1-3b89-43bc-9c99-5748bb75fbc4","Type":"ContainerDied","Data":"53f6186bf9841e4fcdc3943d46575acfe88b954475613bf41508723b3629b40e"} Oct 30 22:27:46.243563 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.243485 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" event={"ID":"d2760ed1-3b89-43bc-9c99-5748bb75fbc4","Type":"ContainerDied","Data":"7fd594b448083d1b5604d943b28f44cf4851527cc6dcc1dbccd998a61b6bfa5e"} Oct 30 22:27:46.243563 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.243509 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fd594b448083d1b5604d943b28f44cf4851527cc6dcc1dbccd998a61b6bfa5e" Oct 30 22:27:46.263955 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.263918 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:27:46.384222 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.384186 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-token\" (UniqueName: \"kubernetes.io/projected/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-token\") pod \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " Oct 30 22:27:46.384740 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.384460 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-credential-socket\") pod \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " Oct 30 22:27:46.384841 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.384788 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-podinfo\") pod \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " Oct 30 22:27:46.384934 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.384900 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-credential-socket" pod "d2760ed1-3b89-43bc-9c99-5748bb75fbc4" (UID: "d2760ed1-3b89-43bc-9c99-5748bb75fbc4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:46.385087 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.385067 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-kube-api-access-kvs49\" (UniqueName: \"kubernetes.io/projected/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-kube-api-access-kvs49\") pod \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " Oct 30 22:27:46.385182 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.385165 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istiod-ca-cert\") pod \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " Oct 30 22:27:46.385365 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.385346 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-workload-certs\") pod \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " Oct 30 22:27:46.385470 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.385454 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-envoy\") pod \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " Oct 30 22:27:46.385527 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.385498 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-workload-socket\") pod \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " Oct 30 22:27:46.385581 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.385526 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-data\") pod \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\" (UID: \"d2760ed1-3b89-43bc-9c99-5748bb75fbc4\") " Oct 30 22:27:46.386814 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.386788 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:46.387031 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.387000 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-data" pod "d2760ed1-3b89-43bc-9c99-5748bb75fbc4" (UID: "d2760ed1-3b89-43bc-9c99-5748bb75fbc4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:46.387893 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.387843 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-workload-certs" pod "d2760ed1-3b89-43bc-9c99-5748bb75fbc4" (UID: "d2760ed1-3b89-43bc-9c99-5748bb75fbc4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:46.388599 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.388543 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-workload-socket" pod "d2760ed1-3b89-43bc-9c99-5748bb75fbc4" (UID: "d2760ed1-3b89-43bc-9c99-5748bb75fbc4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:46.388599 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.388560 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istiod-ca-cert" pod "d2760ed1-3b89-43bc-9c99-5748bb75fbc4" (UID: "d2760ed1-3b89-43bc-9c99-5748bb75fbc4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:46.390417 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.390386 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-kube-api-access-kvs49" pod "d2760ed1-3b89-43bc-9c99-5748bb75fbc4" (UID: "d2760ed1-3b89-43bc-9c99-5748bb75fbc4"). InnerVolumeSpecName "kube-api-access-kvs49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:46.458347 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.446232 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6" podStartSLOduration=2.446208796 podStartE2EDuration="2.446208796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:46.331207192 +0000 UTC m=+2268.883383493" watchObservedRunningTime="2026-10-30 22:27:46.446208796 +0000 UTC m=+2268.998385097" Oct 30 22:27:46.458347 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.451803 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-podinfo" pod "d2760ed1-3b89-43bc-9c99-5748bb75fbc4" (UID: "d2760ed1-3b89-43bc-9c99-5748bb75fbc4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:27:46.458347 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.452419 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-envoy" pod "d2760ed1-3b89-43bc-9c99-5748bb75fbc4" (UID: "d2760ed1-3b89-43bc-9c99-5748bb75fbc4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:46.458347 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.458191 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-token" pod "d2760ed1-3b89-43bc-9c99-5748bb75fbc4" (UID: "d2760ed1-3b89-43bc-9c99-5748bb75fbc4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:46.503274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.495710 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:46.503274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.495756 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:46.503274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.495771 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:46.503274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.495784 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:46.503274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.495797 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:46.503274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.495810 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:46.503274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.495823 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:46.503274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:46.495837 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvs49\" (UniqueName: \"kubernetes.io/projected/d2760ed1-3b89-43bc-9c99-5748bb75fbc4-kube-api-access-kvs49\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:47.268639 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:47.264666 6006 generic.go:356] "Generic (PLEG): container finished" podID="4a0667a5-c67a-46fd-8728-1fc6a950eb58" containerID="d8a42bf2cdd274afb0b5cc456248b26889927fce9c8123cef5b8ae850f7ecc52" exitCode=0 Oct 30 22:27:47.268639 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:47.264811 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5" Oct 30 22:27:47.271502 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:47.271460 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6" event={"ID":"4a0667a5-c67a-46fd-8728-1fc6a950eb58","Type":"ContainerDied","Data":"d8a42bf2cdd274afb0b5cc456248b26889927fce9c8123cef5b8ae850f7ecc52"} Oct 30 22:27:47.891566 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:47.891518 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5"] Oct 30 22:27:47.891865 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:47.891845 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5637a7-openshift-default-5b47575485-gjmv5"] Oct 30 22:27:47.896028 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:47.896005 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk" Oct 30 22:27:48.093807 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:48.093720 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc-maas-api-openshift-ca\") pod \"8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc\" (UID: \"8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc\") " Oct 30 22:27:48.094720 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:48.094697 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc-maas-api-xks-ca\") pod \"8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc\" (UID: \"8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc\") " Oct 30 22:27:48.095277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:48.095244 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc-kube-api-access-cckzx\" (UniqueName: \"kubernetes.io/projected/8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc-kube-api-access-cckzx\") pod \"8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc\" (UID: \"8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc\") " Oct 30 22:27:48.096910 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:48.094309 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc-maas-api-openshift-ca" pod "8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc" (UID: "8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:48.137495 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:48.137455 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc-maas-api-xks-ca" pod "8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc" (UID: "8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:48.137891 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:48.137860 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc-kube-api-access-cckzx" pod "8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc" (UID: "8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc"). InnerVolumeSpecName "kube-api-access-cckzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:48.248466 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:48.248428 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:48.255366 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:48.255320 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:48.255607 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:48.255590 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cckzx\" (UniqueName: \"kubernetes.io/projected/8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc-kube-api-access-cckzx\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:48.288279 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:48.285175 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk" Oct 30 22:27:48.288763 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:48.288271 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk" event={"ID":"8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc","Type":"ContainerDied","Data":"06fad2fef9d13d6dec93b942c94fd789eb58c26d4912756254e71e3d23ab2a54"} Oct 30 22:27:48.288763 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:48.288335 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06fad2fef9d13d6dec93b942c94fd789eb58c26d4912756254e71e3d23ab2a54" Oct 30 22:27:48.699483 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:48.699452 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6" Oct 30 22:27:48.778171 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:48.778067 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a0667a5-c67a-46fd-8728-1fc6a950eb58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0667a5-c67a-46fd-8728-1fc6a950eb58-maas-api-openshift-ca\") pod \"4a0667a5-c67a-46fd-8728-1fc6a950eb58\" (UID: \"4a0667a5-c67a-46fd-8728-1fc6a950eb58\") " Oct 30 22:27:48.778384 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:48.778287 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a0667a5-c67a-46fd-8728-1fc6a950eb58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0667a5-c67a-46fd-8728-1fc6a950eb58-maas-api-xks-ca\") pod \"4a0667a5-c67a-46fd-8728-1fc6a950eb58\" (UID: \"4a0667a5-c67a-46fd-8728-1fc6a950eb58\") " Oct 30 22:27:48.778384 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:48.778343 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a0667a5-c67a-46fd-8728-1fc6a950eb58-kube-api-access-8xqcf\" (UniqueName: \"kubernetes.io/projected/4a0667a5-c67a-46fd-8728-1fc6a950eb58-kube-api-access-8xqcf\") pod \"4a0667a5-c67a-46fd-8728-1fc6a950eb58\" (UID: \"4a0667a5-c67a-46fd-8728-1fc6a950eb58\") " Oct 30 22:27:48.778609 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:48.778581 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a0667a5-c67a-46fd-8728-1fc6a950eb58-maas-api-openshift-ca" pod "4a0667a5-c67a-46fd-8728-1fc6a950eb58" (UID: "4a0667a5-c67a-46fd-8728-1fc6a950eb58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:48.779024 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:48.778991 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0667a5-c67a-46fd-8728-1fc6a950eb58-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:48.780600 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:48.780572 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0667a5-c67a-46fd-8728-1fc6a950eb58-kube-api-access-8xqcf" pod "4a0667a5-c67a-46fd-8728-1fc6a950eb58" (UID: "4a0667a5-c67a-46fd-8728-1fc6a950eb58"). InnerVolumeSpecName "kube-api-access-8xqcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:48.781403 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:48.781382 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a0667a5-c67a-46fd-8728-1fc6a950eb58-maas-api-xks-ca" pod "4a0667a5-c67a-46fd-8728-1fc6a950eb58" (UID: "4a0667a5-c67a-46fd-8728-1fc6a950eb58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:48.880054 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:48.880019 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0667a5-c67a-46fd-8728-1fc6a950eb58-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:48.880054 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:48.880053 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xqcf\" (UniqueName: \"kubernetes.io/projected/4a0667a5-c67a-46fd-8728-1fc6a950eb58-kube-api-access-8xqcf\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:49.294747 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:49.294709 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6" event={"ID":"4a0667a5-c67a-46fd-8728-1fc6a950eb58","Type":"ContainerDied","Data":"c209af8659bb7b0a5a8a4daf630c8ed8a8e4e52d4ebf5cd11b58bf9ac75253d7"} Oct 30 22:27:49.295303 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:49.295283 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c209af8659bb7b0a5a8a4daf630c8ed8a8e4e52d4ebf5cd11b58bf9ac75253d7" Oct 30 22:27:49.295410 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:49.294730 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6" Oct 30 22:27:49.646220 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:49.646129 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2760ed1-3b89-43bc-9c99-5748bb75fbc4" path="/var/lib/kubelet/pods/d2760ed1-3b89-43bc-9c99-5748bb75fbc4/volumes" Oct 30 22:27:52.610555 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:52.610509 6006 patch_prober.go:29] interesting pod/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:27:52.610982 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:52.610591 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" podUID="f9c79304-5ef5-4fc9-ba42-facc6c16f2b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:27:53.586142 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:53.586099 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld"] Oct 30 22:27:53.688056 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:53.688009 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv"] Oct 30 22:27:55.383022 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:55.382988 6006 generic.go:356] "Generic (PLEG): container finished" podID="0ec694be-68a9-4227-96ac-cf7daac3af60" containerID="839dfd2571d546fa1f815a462ff1a8cadf8afee3f65d81846b4a4f7885cf1dcb" exitCode=0 Oct 30 22:27:55.383626 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:55.383066 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" event={"ID":"0ec694be-68a9-4227-96ac-cf7daac3af60","Type":"ContainerDied","Data":"839dfd2571d546fa1f815a462ff1a8cadf8afee3f65d81846b4a4f7885cf1dcb"} Oct 30 22:27:55.383693 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:55.383660 6006 scope.go:122] "RemoveContainer" containerID="839dfd2571d546fa1f815a462ff1a8cadf8afee3f65d81846b4a4f7885cf1dcb" Oct 30 22:27:55.384936 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:55.384916 6006 generic.go:356] "Generic (PLEG): container finished" podID="f9c79304-5ef5-4fc9-ba42-facc6c16f2b0" containerID="b588614d2050bf7f198d97ec8d7765e9acd0373a689feab40d3cd18ba44d2d47" exitCode=0 Oct 30 22:27:55.385057 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:55.385044 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" event={"ID":"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0","Type":"ContainerDied","Data":"b588614d2050bf7f198d97ec8d7765e9acd0373a689feab40d3cd18ba44d2d47"} Oct 30 22:27:55.385398 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:55.385383 6006 scope.go:122] "RemoveContainer" containerID="b588614d2050bf7f198d97ec8d7765e9acd0373a689feab40d3cd18ba44d2d47" Oct 30 22:27:55.385700 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:55.385669 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:27:55.385784 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:55.385760 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:27:55.385849 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:55.385809 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:27:55.387119 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:55.387093 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:27:55.387189 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:55.387156 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:27:55.387230 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:55.387194 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:27:55.612388 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:55.612209 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:27:56.577239 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:56.576239 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" event={"ID":"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0","Type":"ContainerStarted","Data":"f8c67a25f196c2cfb0fc6182c9ad8f31bf9995776e1d2f5226c0a7662cc7973c"} Oct 30 22:27:56.579471 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:56.579429 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" event={"ID":"0ec694be-68a9-4227-96ac-cf7daac3af60","Type":"ContainerStarted","Data":"09efbed3d73532097e2c8cc48b361c1b72d96e02302e6a9927708c8f929606e5"} Oct 30 22:27:57.050132 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:57.049701 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:27:57.050944 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:57.050881 6006 patch_prober.go:29] interesting pod/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 22:27:57.051101 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:57.050989 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" podUID="f9c79304-5ef5-4fc9-ba42-facc6c16f2b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 22:27:57.430425 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:57.422582 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:27:57.452274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:57.436415 6006 patch_prober.go:29] interesting pod/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 22:27:57.452274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:57.436505 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" podUID="0ec694be-68a9-4227-96ac-cf7daac3af60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 22:27:57.971578 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:27:57.971537 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:27:58.048276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:58.045547 6006 patch_prober.go:29] interesting pod/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 22:27:58.048276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:58.045611 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" podUID="f9c79304-5ef5-4fc9-ba42-facc6c16f2b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 22:27:58.427273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:58.420596 6006 patch_prober.go:29] interesting pod/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 22:27:58.427273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:58.420674 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" podUID="0ec694be-68a9-4227-96ac-cf7daac3af60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 22:27:58.587030 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:58.586959 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld" podUID="9e5281f9-05da-4a88-87cd-4852a4edaba7" containerName="payload-processing" containerID="cri-o://01c1bab04f84beb052aadcb6c5309c91370e91c22348dec2387e92aab11a8069" gracePeriod=25 Oct 30 22:27:58.688585 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:58.688353 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv" podUID="de28863a-1f23-489f-aed6-9392cdf58442" containerName="payload-pre-processing" containerID="cri-o://303888e59a3a9f1a73bd7cfd11fdee36ee7d32f754e4abeb2439594ff38dee2b" gracePeriod=25 Oct 30 22:27:59.049571 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:59.043843 6006 patch_prober.go:29] interesting pod/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 22:27:59.049571 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:59.043913 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" podUID="f9c79304-5ef5-4fc9-ba42-facc6c16f2b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 22:27:59.423707 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:59.423663 6006 patch_prober.go:29] interesting pod/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 22:27:59.423979 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:27:59.423945 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" podUID="0ec694be-68a9-4227-96ac-cf7daac3af60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 22:28:00.043763 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:00.043713 6006 patch_prober.go:29] interesting pod/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 22:28:00.044077 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:00.044044 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" podUID="f9c79304-5ef5-4fc9-ba42-facc6c16f2b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 22:28:00.436333 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:00.435389 6006 patch_prober.go:29] interesting pod/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 22:28:00.436333 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:00.435454 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" podUID="0ec694be-68a9-4227-96ac-cf7daac3af60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 22:28:01.049203 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:01.049163 6006 patch_prober.go:29] interesting pod/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 22:28:01.050350 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:01.050310 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" podUID="f9c79304-5ef5-4fc9-ba42-facc6c16f2b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 22:28:01.463016 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:01.462921 6006 patch_prober.go:29] interesting pod/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 22:28:01.463659 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:01.463626 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" podUID="0ec694be-68a9-4227-96ac-cf7daac3af60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 22:28:02.049577 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:02.049540 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:28:02.050055 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:02.050027 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:28:02.050994 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:02.050970 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:28:02.427573 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:02.427473 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:28:02.429859 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:02.429831 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:28:02.431741 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:02.431712 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:28:07.707966 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:07.707917 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv" podUID="de28863a-1f23-489f-aed6-9392cdf58442" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:28:07.708423 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:07.707930 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld" podUID="9e5281f9-05da-4a88-87cd-4852a4edaba7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:28:13.308345 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:13.308245 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:28:17.727276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:17.726387 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld" podUID="9e5281f9-05da-4a88-87cd-4852a4edaba7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:28:17.727276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:17.726957 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv" podUID="de28863a-1f23-489f-aed6-9392cdf58442" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:28:18.854222 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:18.853912 6006 generic.go:356] "Generic (PLEG): container finished" podID="9e5281f9-05da-4a88-87cd-4852a4edaba7" containerID="01c1bab04f84beb052aadcb6c5309c91370e91c22348dec2387e92aab11a8069" exitCode=0 Oct 30 22:28:18.854222 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:18.854105 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld" event={"ID":"9e5281f9-05da-4a88-87cd-4852a4edaba7","Type":"ContainerDied","Data":"01c1bab04f84beb052aadcb6c5309c91370e91c22348dec2387e92aab11a8069"} Oct 30 22:28:18.856934 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:18.856833 6006 generic.go:356] "Generic (PLEG): container finished" podID="de28863a-1f23-489f-aed6-9392cdf58442" containerID="303888e59a3a9f1a73bd7cfd11fdee36ee7d32f754e4abeb2439594ff38dee2b" exitCode=0 Oct 30 22:28:18.856934 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:18.856875 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv" event={"ID":"de28863a-1f23-489f-aed6-9392cdf58442","Type":"ContainerDied","Data":"303888e59a3a9f1a73bd7cfd11fdee36ee7d32f754e4abeb2439594ff38dee2b"} Oct 30 22:28:18.941054 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:18.939909 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn"] Oct 30 22:28:18.941539 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:18.941497 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn" podUID="0583d379-6fbe-403b-bf73-2dca5514c792" containerName="maas-api" containerID="cri-o://73abe01db0c06a3a42c0def225e25e7b6e7ce9308091fdf87f772708b0a8a05d" gracePeriod=30 Oct 30 22:28:19.044125 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:19.044084 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld" Oct 30 22:28:19.166335 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:19.148873 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e5281f9-05da-4a88-87cd-4852a4edaba7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5281f9-05da-4a88-87cd-4852a4edaba7-plugins-config-volume\") pod \"9e5281f9-05da-4a88-87cd-4852a4edaba7\" (UID: \"9e5281f9-05da-4a88-87cd-4852a4edaba7\") " Oct 30 22:28:19.166335 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:19.148959 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e5281f9-05da-4a88-87cd-4852a4edaba7-kube-api-access-k2fg2\" (UniqueName: \"kubernetes.io/projected/9e5281f9-05da-4a88-87cd-4852a4edaba7-kube-api-access-k2fg2\") pod \"9e5281f9-05da-4a88-87cd-4852a4edaba7\" (UID: \"9e5281f9-05da-4a88-87cd-4852a4edaba7\") " Oct 30 22:28:19.166335 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:19.163958 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5281f9-05da-4a88-87cd-4852a4edaba7-kube-api-access-k2fg2" pod "9e5281f9-05da-4a88-87cd-4852a4edaba7" (UID: "9e5281f9-05da-4a88-87cd-4852a4edaba7"). InnerVolumeSpecName "kube-api-access-k2fg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:19.166335 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:19.164381 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5281f9-05da-4a88-87cd-4852a4edaba7-plugins-config-volume" pod "9e5281f9-05da-4a88-87cd-4852a4edaba7" (UID: "9e5281f9-05da-4a88-87cd-4852a4edaba7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:19.254317 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:19.253751 6006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5281f9-05da-4a88-87cd-4852a4edaba7-plugins-config-volume\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:19.254317 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:19.253787 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2fg2\" (UniqueName: \"kubernetes.io/projected/9e5281f9-05da-4a88-87cd-4852a4edaba7-kube-api-access-k2fg2\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:19.275223 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:19.274826 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv" Oct 30 22:28:19.363278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:19.354330 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de28863a-1f23-489f-aed6-9392cdf58442-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de28863a-1f23-489f-aed6-9392cdf58442-plugins-config-volume\") pod \"de28863a-1f23-489f-aed6-9392cdf58442\" (UID: \"de28863a-1f23-489f-aed6-9392cdf58442\") " Oct 30 22:28:19.363278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:19.354623 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de28863a-1f23-489f-aed6-9392cdf58442-kube-api-access-pxprq\" (UniqueName: \"kubernetes.io/projected/de28863a-1f23-489f-aed6-9392cdf58442-kube-api-access-pxprq\") pod \"de28863a-1f23-489f-aed6-9392cdf58442\" (UID: \"de28863a-1f23-489f-aed6-9392cdf58442\") " Oct 30 22:28:19.383558 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:19.383512 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de28863a-1f23-489f-aed6-9392cdf58442-plugins-config-volume" pod "de28863a-1f23-489f-aed6-9392cdf58442" (UID: "de28863a-1f23-489f-aed6-9392cdf58442"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:19.473730 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:19.473617 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de28863a-1f23-489f-aed6-9392cdf58442-kube-api-access-pxprq" pod "de28863a-1f23-489f-aed6-9392cdf58442" (UID: "de28863a-1f23-489f-aed6-9392cdf58442"). InnerVolumeSpecName "kube-api-access-pxprq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:19.496666 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:19.477320 6006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de28863a-1f23-489f-aed6-9392cdf58442-plugins-config-volume\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:19.496666 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:19.477356 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxprq\" (UniqueName: \"kubernetes.io/projected/de28863a-1f23-489f-aed6-9392cdf58442-kube-api-access-pxprq\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:19.792586 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:19.792552 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn" Oct 30 22:28:19.911286 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:19.909040 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0583d379-6fbe-403b-bf73-2dca5514c792-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0583d379-6fbe-403b-bf73-2dca5514c792-metrics-certs\") pod \"0583d379-6fbe-403b-bf73-2dca5514c792\" (UID: \"0583d379-6fbe-403b-bf73-2dca5514c792\") " Oct 30 22:28:19.911286 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:19.909130 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0583d379-6fbe-403b-bf73-2dca5514c792-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0583d379-6fbe-403b-bf73-2dca5514c792-maas-api-tls\") pod \"0583d379-6fbe-403b-bf73-2dca5514c792\" (UID: \"0583d379-6fbe-403b-bf73-2dca5514c792\") " Oct 30 22:28:19.911286 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:19.909212 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0583d379-6fbe-403b-bf73-2dca5514c792-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0583d379-6fbe-403b-bf73-2dca5514c792-openshift-service-ca\") pod \"0583d379-6fbe-403b-bf73-2dca5514c792\" (UID: \"0583d379-6fbe-403b-bf73-2dca5514c792\") " Oct 30 22:28:19.911286 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:19.909242 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0583d379-6fbe-403b-bf73-2dca5514c792-kube-api-access-fz8fz\" (UniqueName: \"kubernetes.io/projected/0583d379-6fbe-403b-bf73-2dca5514c792-kube-api-access-fz8fz\") pod \"0583d379-6fbe-403b-bf73-2dca5514c792\" (UID: \"0583d379-6fbe-403b-bf73-2dca5514c792\") " Oct 30 22:28:19.915309 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:19.913395 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0583d379-6fbe-403b-bf73-2dca5514c792-openshift-service-ca" pod "0583d379-6fbe-403b-bf73-2dca5514c792" (UID: "0583d379-6fbe-403b-bf73-2dca5514c792"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:19.964273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:19.957052 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0583d379-6fbe-403b-bf73-2dca5514c792-metrics-certs" pod "0583d379-6fbe-403b-bf73-2dca5514c792" (UID: "0583d379-6fbe-403b-bf73-2dca5514c792"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:19.964273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:19.957538 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0583d379-6fbe-403b-bf73-2dca5514c792-maas-api-tls" pod "0583d379-6fbe-403b-bf73-2dca5514c792" (UID: "0583d379-6fbe-403b-bf73-2dca5514c792"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:19.985504 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:19.982441 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0583d379-6fbe-403b-bf73-2dca5514c792-kube-api-access-fz8fz" pod "0583d379-6fbe-403b-bf73-2dca5514c792" (UID: "0583d379-6fbe-403b-bf73-2dca5514c792"). InnerVolumeSpecName "kube-api-access-fz8fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:20.078333 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:20.075804 6006 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0583d379-6fbe-403b-bf73-2dca5514c792-metrics-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:20.078333 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:20.075840 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0583d379-6fbe-403b-bf73-2dca5514c792-maas-api-tls\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:20.078333 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:20.075857 6006 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0583d379-6fbe-403b-bf73-2dca5514c792-openshift-service-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:20.078333 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:20.075872 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz8fz\" (UniqueName: \"kubernetes.io/projected/0583d379-6fbe-403b-bf73-2dca5514c792-kube-api-access-fz8fz\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:20.106478 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:20.102637 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld" event={"ID":"9e5281f9-05da-4a88-87cd-4852a4edaba7","Type":"ContainerDied","Data":"c45eb72a33693317d001cc8ed800264c39d81bbbd9eb2668bf1133f1474e2260"} Oct 30 22:28:20.106478 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:20.102715 6006 scope.go:122] "RemoveContainer" containerID="01c1bab04f84beb052aadcb6c5309c91370e91c22348dec2387e92aab11a8069" Oct 30 22:28:20.106478 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:20.102892 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld" Oct 30 22:28:20.114271 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:20.107516 6006 generic.go:356] "Generic (PLEG): container finished" podID="0583d379-6fbe-403b-bf73-2dca5514c792" containerID="73abe01db0c06a3a42c0def225e25e7b6e7ce9308091fdf87f772708b0a8a05d" exitCode=0 Oct 30 22:28:20.114271 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:20.107709 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn" event={"ID":"0583d379-6fbe-403b-bf73-2dca5514c792","Type":"ContainerDied","Data":"73abe01db0c06a3a42c0def225e25e7b6e7ce9308091fdf87f772708b0a8a05d"} Oct 30 22:28:20.114271 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:20.107761 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn" event={"ID":"0583d379-6fbe-403b-bf73-2dca5514c792","Type":"ContainerDied","Data":"c3cadeb89f370c94595a531cfec3029009d3a31ef4c958d75910379c2d95f8fa"} Oct 30 22:28:20.114271 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:20.107848 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn" Oct 30 22:28:20.125815 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:20.125776 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv" event={"ID":"de28863a-1f23-489f-aed6-9392cdf58442","Type":"ContainerDied","Data":"e4c657b094d3f13df1482a5bf15c8e5bb3cb74757ec5545d544426c9591ba66b"} Oct 30 22:28:20.126155 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:20.126090 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv" Oct 30 22:28:20.158211 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:20.158101 6006 scope.go:122] "RemoveContainer" containerID="73abe01db0c06a3a42c0def225e25e7b6e7ce9308091fdf87f772708b0a8a05d" Oct 30 22:28:20.172711 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:20.172680 6006 scope.go:122] "RemoveContainer" containerID="73abe01db0c06a3a42c0def225e25e7b6e7ce9308091fdf87f772708b0a8a05d" Oct 30 22:28:20.173094 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:28:20.173067 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73abe01db0c06a3a42c0def225e25e7b6e7ce9308091fdf87f772708b0a8a05d\": container with ID starting with 73abe01db0c06a3a42c0def225e25e7b6e7ce9308091fdf87f772708b0a8a05d not found: ID does not exist" containerID="73abe01db0c06a3a42c0def225e25e7b6e7ce9308091fdf87f772708b0a8a05d" Oct 30 22:28:20.173159 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:20.173107 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73abe01db0c06a3a42c0def225e25e7b6e7ce9308091fdf87f772708b0a8a05d"} err="failed to get container status \"73abe01db0c06a3a42c0def225e25e7b6e7ce9308091fdf87f772708b0a8a05d\": rpc error: code = NotFound desc = could not find container \"73abe01db0c06a3a42c0def225e25e7b6e7ce9308091fdf87f772708b0a8a05d\": container with ID starting with 73abe01db0c06a3a42c0def225e25e7b6e7ce9308091fdf87f772708b0a8a05d not found: ID does not exist" Oct 30 22:28:20.173159 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:20.173132 6006 scope.go:122] "RemoveContainer" containerID="303888e59a3a9f1a73bd7cfd11fdee36ee7d32f754e4abeb2439594ff38dee2b" Oct 30 22:28:20.267104 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:20.267049 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld"] Oct 30 22:28:20.271142 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:20.271109 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-7bb1c2-7bc7b7579-6f7ld"] Oct 30 22:28:20.377425 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:20.377325 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv"] Oct 30 22:28:20.380586 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:20.380557 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7bb1c2-6fdd9dbbbc-799lv"] Oct 30 22:28:20.481909 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:20.481860 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn"] Oct 30 22:28:20.528179 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:20.528144 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c24d61-54688689c7-qzwgn"] Oct 30 22:28:21.039297 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:21.039229 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6"] Oct 30 22:28:21.043070 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:21.043030 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c24d61-4kqs6"] Oct 30 22:28:21.605554 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:21.605516 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0583d379-6fbe-403b-bf73-2dca5514c792" path="/var/lib/kubelet/pods/0583d379-6fbe-403b-bf73-2dca5514c792/volumes" Oct 30 22:28:21.606093 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:21.606069 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e5281f9-05da-4a88-87cd-4852a4edaba7" path="/var/lib/kubelet/pods/9e5281f9-05da-4a88-87cd-4852a4edaba7/volumes" Oct 30 22:28:21.606594 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:21.606574 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f37fb44-2790-4b49-b217-64873b88039b" path="/var/lib/kubelet/pods/9f37fb44-2790-4b49-b217-64873b88039b/volumes" Oct 30 22:28:21.607355 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:21.607328 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de28863a-1f23-489f-aed6-9392cdf58442" path="/var/lib/kubelet/pods/de28863a-1f23-489f-aed6-9392cdf58442/volumes" Oct 30 22:28:22.923307 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:22.923235 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf"] Oct 30 22:28:22.923935 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:22.923917 6006 cpu_manager.go:414] "containerMap: removing container" podUID="de28863a-1f23-489f-aed6-9392cdf58442" containerName="payload-pre-processing" Oct 30 22:28:22.923935 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:22.923934 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de28863a-1f23-489f-aed6-9392cdf58442" containerName="payload-pre-processing" Oct 30 22:28:22.924078 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:22.923954 6006 cpu_manager.go:414] "containerMap: removing container" podUID="d2760ed1-3b89-43bc-9c99-5748bb75fbc4" containerName="istio-proxy" Oct 30 22:28:22.924078 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:22.923962 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2760ed1-3b89-43bc-9c99-5748bb75fbc4" containerName="istio-proxy" Oct 30 22:28:22.924078 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:22.923972 6006 cpu_manager.go:414] "containerMap: removing container" podUID="9e5281f9-05da-4a88-87cd-4852a4edaba7" containerName="payload-processing" Oct 30 22:28:22.924078 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:22.923977 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e5281f9-05da-4a88-87cd-4852a4edaba7" containerName="payload-processing" Oct 30 22:28:22.924078 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:22.923996 6006 cpu_manager.go:414] "containerMap: removing container" podUID="0583d379-6fbe-403b-bf73-2dca5514c792" containerName="maas-api" Oct 30 22:28:22.924078 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:22.924000 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0583d379-6fbe-403b-bf73-2dca5514c792" containerName="maas-api" Oct 30 22:28:22.924078 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:22.924017 6006 cpu_manager.go:414] "containerMap: removing container" podUID="8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc" containerName="revoke-keys" Oct 30 22:28:22.924078 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:22.924022 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc" containerName="revoke-keys" Oct 30 22:28:22.924078 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:22.924066 6006 cpu_manager.go:414] "containerMap: removing container" podUID="4a0667a5-c67a-46fd-8728-1fc6a950eb58" containerName="revoke-keys" Oct 30 22:28:22.924078 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:22.924072 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a0667a5-c67a-46fd-8728-1fc6a950eb58" containerName="revoke-keys" Oct 30 22:28:22.924783 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:22.924176 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc" containerName="revoke-keys" Oct 30 22:28:22.924783 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:22.924192 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a0667a5-c67a-46fd-8728-1fc6a950eb58" containerName="revoke-keys" Oct 30 22:28:22.924783 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:22.924203 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="de28863a-1f23-489f-aed6-9392cdf58442" containerName="payload-pre-processing" Oct 30 22:28:22.924783 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:22.924210 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2760ed1-3b89-43bc-9c99-5748bb75fbc4" containerName="istio-proxy" Oct 30 22:28:22.924783 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:22.924219 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e5281f9-05da-4a88-87cd-4852a4edaba7" containerName="payload-processing" Oct 30 22:28:22.924783 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:22.924228 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="0583d379-6fbe-403b-bf73-2dca5514c792" containerName="maas-api" Oct 30 22:28:22.930992 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:22.930971 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf" Oct 30 22:28:22.984580 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:22.984546 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf"] Oct 30 22:28:23.019008 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:23.018964 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abbf73e3-26bb-4927-91b7-c0ece65c8fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf\" (UID: \"abbf73e3-26bb-4927-91b7-c0ece65c8fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf" Oct 30 22:28:23.019210 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:23.019045 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5dcg\" (UniqueName: \"kubernetes.io/projected/abbf73e3-26bb-4927-91b7-c0ece65c8fbb-kube-api-access-d5dcg\") pod \"maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf\" (UID: \"abbf73e3-26bb-4927-91b7-c0ece65c8fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf" Oct 30 22:28:23.019210 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:23.019199 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abbf73e3-26bb-4927-91b7-c0ece65c8fbb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf\" (UID: \"abbf73e3-26bb-4927-91b7-c0ece65c8fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf" Oct 30 22:28:23.026540 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:23.026498 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:28:23.026540 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:23.026517 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:28:23.120439 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:23.120401 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abbf73e3-26bb-4927-91b7-c0ece65c8fbb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf\" (UID: \"abbf73e3-26bb-4927-91b7-c0ece65c8fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf" Oct 30 22:28:23.120640 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:23.120540 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abbf73e3-26bb-4927-91b7-c0ece65c8fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf\" (UID: \"abbf73e3-26bb-4927-91b7-c0ece65c8fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf" Oct 30 22:28:23.120640 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:23.120627 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5dcg\" (UniqueName: \"kubernetes.io/projected/abbf73e3-26bb-4927-91b7-c0ece65c8fbb-kube-api-access-d5dcg\") pod \"maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf\" (UID: \"abbf73e3-26bb-4927-91b7-c0ece65c8fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf" Oct 30 22:28:23.121343 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:23.121319 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abbf73e3-26bb-4927-91b7-c0ece65c8fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf\" (UID: \"abbf73e3-26bb-4927-91b7-c0ece65c8fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf" Oct 30 22:28:23.122958 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:23.122938 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abbf73e3-26bb-4927-91b7-c0ece65c8fbb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf\" (UID: \"abbf73e3-26bb-4927-91b7-c0ece65c8fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf" Oct 30 22:28:23.179567 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:23.179496 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5dcg\" (UniqueName: \"kubernetes.io/projected/abbf73e3-26bb-4927-91b7-c0ece65c8fbb-kube-api-access-d5dcg\") pod \"maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf\" (UID: \"abbf73e3-26bb-4927-91b7-c0ece65c8fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf" Oct 30 22:28:23.249913 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:23.249866 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf" Oct 30 22:28:23.630927 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:23.630885 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf"] Oct 30 22:28:23.635196 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:28:23.635162 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabbf73e3_26bb_4927_91b7_c0ece65c8fbb.slice/crio-867111bca2a67d64cf90f113bcaf254b8e8e228ecaa9da681bb704e896f591df WatchSource:0}: Error finding container 867111bca2a67d64cf90f113bcaf254b8e8e228ecaa9da681bb704e896f591df: Status 404 returned error can't find the container with id 867111bca2a67d64cf90f113bcaf254b8e8e228ecaa9da681bb704e896f591df Oct 30 22:28:24.165157 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:24.165114 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf" event={"ID":"abbf73e3-26bb-4927-91b7-c0ece65c8fbb","Type":"ContainerStarted","Data":"935ed079f2936214634d3659d7a6e4361fc3f04d0ec53ffa7bd410e62dea7dee"} Oct 30 22:28:24.165691 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:24.165188 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf" event={"ID":"abbf73e3-26bb-4927-91b7-c0ece65c8fbb","Type":"ContainerStarted","Data":"867111bca2a67d64cf90f113bcaf254b8e8e228ecaa9da681bb704e896f591df"} Oct 30 22:28:24.276868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:24.276753 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf" podStartSLOduration=2.2767388410000002 podStartE2EDuration="2.276738841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:24.274830739 +0000 UTC m=+2306.827007040" watchObservedRunningTime="2026-10-30 22:28:24.276738841 +0000 UTC m=+2306.828915141" Oct 30 22:28:25.177618 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:25.177584 6006 generic.go:356] "Generic (PLEG): container finished" podID="abbf73e3-26bb-4927-91b7-c0ece65c8fbb" containerID="935ed079f2936214634d3659d7a6e4361fc3f04d0ec53ffa7bd410e62dea7dee" exitCode=0 Oct 30 22:28:25.178056 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:25.177815 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf" event={"ID":"abbf73e3-26bb-4927-91b7-c0ece65c8fbb","Type":"ContainerDied","Data":"935ed079f2936214634d3659d7a6e4361fc3f04d0ec53ffa7bd410e62dea7dee"} Oct 30 22:28:25.179809 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:25.179781 6006 generic.go:356] "Generic (PLEG): container finished" podID="e020ac39-b5d1-4657-9b2b-ce1a06e4c988" containerID="989bcae7013c3d8510aa0f5888ff3ed850c95eaad116bc27c5d9bb33ce63ed01" exitCode=0 Oct 30 22:28:25.179934 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:25.179849 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" event={"ID":"e020ac39-b5d1-4657-9b2b-ce1a06e4c988","Type":"ContainerDied","Data":"989bcae7013c3d8510aa0f5888ff3ed850c95eaad116bc27c5d9bb33ce63ed01"} Oct 30 22:28:25.180299 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:25.180282 6006 scope.go:122] "RemoveContainer" containerID="989bcae7013c3d8510aa0f5888ff3ed850c95eaad116bc27c5d9bb33ce63ed01" Oct 30 22:28:25.182060 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:25.182030 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:28:25.182150 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:25.182100 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:28:25.182150 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:25.182131 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:28:26.190051 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:26.190013 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" event={"ID":"e020ac39-b5d1-4657-9b2b-ce1a06e4c988","Type":"ContainerStarted","Data":"608b0c0a3f78a7055564dc3582544787e187a2f6626774880c141336c61340b6"} Oct 30 22:28:26.436491 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:26.436459 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf" Oct 30 22:28:26.483885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:26.482093 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abbf73e3-26bb-4927-91b7-c0ece65c8fbb-kube-api-access-d5dcg\" (UniqueName: \"kubernetes.io/projected/abbf73e3-26bb-4927-91b7-c0ece65c8fbb-kube-api-access-d5dcg\") pod \"abbf73e3-26bb-4927-91b7-c0ece65c8fbb\" (UID: \"abbf73e3-26bb-4927-91b7-c0ece65c8fbb\") " Oct 30 22:28:26.483885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:26.482220 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abbf73e3-26bb-4927-91b7-c0ece65c8fbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abbf73e3-26bb-4927-91b7-c0ece65c8fbb-maas-api-openshift-ca\") pod \"abbf73e3-26bb-4927-91b7-c0ece65c8fbb\" (UID: \"abbf73e3-26bb-4927-91b7-c0ece65c8fbb\") " Oct 30 22:28:26.483885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:26.482431 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abbf73e3-26bb-4927-91b7-c0ece65c8fbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abbf73e3-26bb-4927-91b7-c0ece65c8fbb-maas-api-xks-ca\") pod \"abbf73e3-26bb-4927-91b7-c0ece65c8fbb\" (UID: \"abbf73e3-26bb-4927-91b7-c0ece65c8fbb\") " Oct 30 22:28:26.484510 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:26.484471 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abbf73e3-26bb-4927-91b7-c0ece65c8fbb-maas-api-openshift-ca" pod "abbf73e3-26bb-4927-91b7-c0ece65c8fbb" (UID: "abbf73e3-26bb-4927-91b7-c0ece65c8fbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:26.517389 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:26.508686 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abbf73e3-26bb-4927-91b7-c0ece65c8fbb-kube-api-access-d5dcg" pod "abbf73e3-26bb-4927-91b7-c0ece65c8fbb" (UID: "abbf73e3-26bb-4927-91b7-c0ece65c8fbb"). InnerVolumeSpecName "kube-api-access-d5dcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:26.517389 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:26.511387 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abbf73e3-26bb-4927-91b7-c0ece65c8fbb-maas-api-xks-ca" pod "abbf73e3-26bb-4927-91b7-c0ece65c8fbb" (UID: "abbf73e3-26bb-4927-91b7-c0ece65c8fbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:26.587374 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:26.586827 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abbf73e3-26bb-4927-91b7-c0ece65c8fbb-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:26.587374 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:26.586876 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abbf73e3-26bb-4927-91b7-c0ece65c8fbb-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:26.587374 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:26.586890 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5dcg\" (UniqueName: \"kubernetes.io/projected/abbf73e3-26bb-4927-91b7-c0ece65c8fbb-kube-api-access-d5dcg\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:26.677596 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:26.677556 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:28:26.678840 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:26.678807 6006 patch_prober.go:29] interesting pod/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 22:28:26.678963 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:26.678864 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" podUID="e020ac39-b5d1-4657-9b2b-ce1a06e4c988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 22:28:26.976875 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:26.976837 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg"] Oct 30 22:28:26.977565 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:26.977543 6006 cpu_manager.go:414] "containerMap: removing container" podUID="abbf73e3-26bb-4927-91b7-c0ece65c8fbb" containerName="revoke-keys" Oct 30 22:28:26.977691 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:26.977565 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abbf73e3-26bb-4927-91b7-c0ece65c8fbb" containerName="revoke-keys" Oct 30 22:28:26.977789 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:26.977775 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="abbf73e3-26bb-4927-91b7-c0ece65c8fbb" containerName="revoke-keys" Oct 30 22:28:26.981678 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:26.981651 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg" Oct 30 22:28:26.991769 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:26.991746 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b794876-ccb6-42a8-bdb9-7f35d3e97803-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg\" (UID: \"3b794876-ccb6-42a8-bdb9-7f35d3e97803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg" Oct 30 22:28:26.991885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:26.991870 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd9zl\" (UniqueName: \"kubernetes.io/projected/3b794876-ccb6-42a8-bdb9-7f35d3e97803-kube-api-access-qd9zl\") pod \"maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg\" (UID: \"3b794876-ccb6-42a8-bdb9-7f35d3e97803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg" Oct 30 22:28:26.991951 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:26.991928 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b794876-ccb6-42a8-bdb9-7f35d3e97803-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg\" (UID: \"3b794876-ccb6-42a8-bdb9-7f35d3e97803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg" Oct 30 22:28:27.044314 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:27.044273 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg"] Oct 30 22:28:27.093400 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:27.093361 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd9zl\" (UniqueName: \"kubernetes.io/projected/3b794876-ccb6-42a8-bdb9-7f35d3e97803-kube-api-access-qd9zl\") pod \"maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg\" (UID: \"3b794876-ccb6-42a8-bdb9-7f35d3e97803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg" Oct 30 22:28:27.093598 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:27.093459 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b794876-ccb6-42a8-bdb9-7f35d3e97803-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg\" (UID: \"3b794876-ccb6-42a8-bdb9-7f35d3e97803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg" Oct 30 22:28:27.093598 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:27.093499 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b794876-ccb6-42a8-bdb9-7f35d3e97803-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg\" (UID: \"3b794876-ccb6-42a8-bdb9-7f35d3e97803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg" Oct 30 22:28:27.094376 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:27.094310 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b794876-ccb6-42a8-bdb9-7f35d3e97803-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg\" (UID: \"3b794876-ccb6-42a8-bdb9-7f35d3e97803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg" Oct 30 22:28:27.096623 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:27.096593 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b794876-ccb6-42a8-bdb9-7f35d3e97803-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg\" (UID: \"3b794876-ccb6-42a8-bdb9-7f35d3e97803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg" Oct 30 22:28:27.193577 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:27.193543 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd9zl\" (UniqueName: \"kubernetes.io/projected/3b794876-ccb6-42a8-bdb9-7f35d3e97803-kube-api-access-qd9zl\") pod \"maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg\" (UID: \"3b794876-ccb6-42a8-bdb9-7f35d3e97803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg" Oct 30 22:28:27.237342 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:27.223024 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf" Oct 30 22:28:27.239484 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:27.239434 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf" event={"ID":"abbf73e3-26bb-4927-91b7-c0ece65c8fbb","Type":"ContainerDied","Data":"867111bca2a67d64cf90f113bcaf254b8e8e228ecaa9da681bb704e896f591df"} Oct 30 22:28:27.239753 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:27.239695 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="867111bca2a67d64cf90f113bcaf254b8e8e228ecaa9da681bb704e896f591df" Oct 30 22:28:27.323541 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:27.323474 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg" Oct 30 22:28:27.678007 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:27.677968 6006 patch_prober.go:29] interesting pod/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 22:28:27.678161 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:27.678024 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" podUID="e020ac39-b5d1-4657-9b2b-ce1a06e4c988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 22:28:27.728325 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:27.728277 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg"] Oct 30 22:28:27.729710 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:28:27.729669 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b794876_ccb6_42a8_bdb9_7f35d3e97803.slice/crio-341803e474439ad8f9db04d2eec7bb921c5c9150a3a424e753b9fd2b0f2a50a5 WatchSource:0}: Error finding container 341803e474439ad8f9db04d2eec7bb921c5c9150a3a424e753b9fd2b0f2a50a5: Status 404 returned error can't find the container with id 341803e474439ad8f9db04d2eec7bb921c5c9150a3a424e753b9fd2b0f2a50a5 Oct 30 22:28:27.951648 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:28:27.951554 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:28:28.030094 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:28.030060 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:28:28.236287 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:28.236217 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg" event={"ID":"3b794876-ccb6-42a8-bdb9-7f35d3e97803","Type":"ContainerStarted","Data":"341803e474439ad8f9db04d2eec7bb921c5c9150a3a424e753b9fd2b0f2a50a5"} Oct 30 22:28:28.679777 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:28.679741 6006 patch_prober.go:29] interesting pod/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 22:28:28.680085 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:28.680047 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" podUID="e020ac39-b5d1-4657-9b2b-ce1a06e4c988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 22:28:29.245513 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:29.245424 6006 generic.go:356] "Generic (PLEG): container finished" podID="3b794876-ccb6-42a8-bdb9-7f35d3e97803" containerID="7e7c601a5291b50a4df14a810e987247fb56aad07b733680353e806077b931ff" exitCode=0 Oct 30 22:28:29.245899 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:29.245508 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg" event={"ID":"3b794876-ccb6-42a8-bdb9-7f35d3e97803","Type":"ContainerDied","Data":"7e7c601a5291b50a4df14a810e987247fb56aad07b733680353e806077b931ff"} Oct 30 22:28:29.700654 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:29.700617 6006 patch_prober.go:29] interesting pod/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 22:28:29.715275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:29.700919 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" podUID="e020ac39-b5d1-4657-9b2b-ce1a06e4c988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 22:28:30.475972 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.475942 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg" Oct 30 22:28:30.530325 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.527822 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b794876-ccb6-42a8-bdb9-7f35d3e97803-kube-api-access-qd9zl\" (UniqueName: \"kubernetes.io/projected/3b794876-ccb6-42a8-bdb9-7f35d3e97803-kube-api-access-qd9zl\") pod \"3b794876-ccb6-42a8-bdb9-7f35d3e97803\" (UID: \"3b794876-ccb6-42a8-bdb9-7f35d3e97803\") " Oct 30 22:28:30.530325 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.527936 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b794876-ccb6-42a8-bdb9-7f35d3e97803-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b794876-ccb6-42a8-bdb9-7f35d3e97803-maas-api-openshift-ca\") pod \"3b794876-ccb6-42a8-bdb9-7f35d3e97803\" (UID: \"3b794876-ccb6-42a8-bdb9-7f35d3e97803\") " Oct 30 22:28:30.530325 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.528191 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b794876-ccb6-42a8-bdb9-7f35d3e97803-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b794876-ccb6-42a8-bdb9-7f35d3e97803-maas-api-xks-ca\") pod \"3b794876-ccb6-42a8-bdb9-7f35d3e97803\" (UID: \"3b794876-ccb6-42a8-bdb9-7f35d3e97803\") " Oct 30 22:28:30.530325 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.529832 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b794876-ccb6-42a8-bdb9-7f35d3e97803-maas-api-openshift-ca" pod "3b794876-ccb6-42a8-bdb9-7f35d3e97803" (UID: "3b794876-ccb6-42a8-bdb9-7f35d3e97803"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:30.532330 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.531753 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b794876-ccb6-42a8-bdb9-7f35d3e97803-maas-api-xks-ca" pod "3b794876-ccb6-42a8-bdb9-7f35d3e97803" (UID: "3b794876-ccb6-42a8-bdb9-7f35d3e97803"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:30.532832 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.532604 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33c6d78b-pdh26"] Oct 30 22:28:30.533606 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.533582 6006 cpu_manager.go:414] "containerMap: removing container" podUID="3b794876-ccb6-42a8-bdb9-7f35d3e97803" containerName="revoke-keys" Oct 30 22:28:30.533606 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.533606 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b794876-ccb6-42a8-bdb9-7f35d3e97803" containerName="revoke-keys" Oct 30 22:28:30.533757 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.533735 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b794876-ccb6-42a8-bdb9-7f35d3e97803-kube-api-access-qd9zl" pod "3b794876-ccb6-42a8-bdb9-7f35d3e97803" (UID: "3b794876-ccb6-42a8-bdb9-7f35d3e97803"). InnerVolumeSpecName "kube-api-access-qd9zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:30.533863 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.533847 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b794876-ccb6-42a8-bdb9-7f35d3e97803" containerName="revoke-keys" Oct 30 22:28:30.537475 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.537454 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c6d78b-pdh26" Oct 30 22:28:30.559661 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.559626 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0feb5925-f7fh9"] Oct 30 22:28:30.564286 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.564262 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feb5925-f7fh9" Oct 30 22:28:30.584337 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.584307 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33c6d78b-pdh26"] Oct 30 22:28:30.636278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.636174 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3094773b-d617-49d9-ac6a-435bd081ff21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0feb5925-f7fh9\" (UID: \"3094773b-d617-49d9-ac6a-435bd081ff21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feb5925-f7fh9" Oct 30 22:28:30.636460 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.636337 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33c6d78b-pdh26\" (UID: \"dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c6d78b-pdh26" Oct 30 22:28:30.636460 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.636452 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33c6d78b-pdh26\" (UID: \"dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c6d78b-pdh26" Oct 30 22:28:30.636596 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.636500 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhl6k\" (UniqueName: \"kubernetes.io/projected/3094773b-d617-49d9-ac6a-435bd081ff21-kube-api-access-dhl6k\") pod \"maas-api-revoke-sub-0feb5925-f7fh9\" (UID: \"3094773b-d617-49d9-ac6a-435bd081ff21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feb5925-f7fh9" Oct 30 22:28:30.636596 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.636536 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcpq8\" (UniqueName: \"kubernetes.io/projected/dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26-kube-api-access-vcpq8\") pod \"maas-api-revoke-sub-33c6d78b-pdh26\" (UID: \"dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c6d78b-pdh26" Oct 30 22:28:30.636717 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.636692 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3094773b-d617-49d9-ac6a-435bd081ff21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0feb5925-f7fh9\" (UID: \"3094773b-d617-49d9-ac6a-435bd081ff21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feb5925-f7fh9" Oct 30 22:28:30.637704 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.636803 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd9zl\" (UniqueName: \"kubernetes.io/projected/3b794876-ccb6-42a8-bdb9-7f35d3e97803-kube-api-access-qd9zl\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:30.637704 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.636825 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b794876-ccb6-42a8-bdb9-7f35d3e97803-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:30.637704 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.636884 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b794876-ccb6-42a8-bdb9-7f35d3e97803-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:30.647794 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.646296 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0feb5925-f7fh9"] Oct 30 22:28:30.677445 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.677395 6006 patch_prober.go:29] interesting pod/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 22:28:30.677613 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.677464 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" podUID="e020ac39-b5d1-4657-9b2b-ce1a06e4c988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 22:28:30.737725 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.737696 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhl6k\" (UniqueName: \"kubernetes.io/projected/3094773b-d617-49d9-ac6a-435bd081ff21-kube-api-access-dhl6k\") pod \"maas-api-revoke-sub-0feb5925-f7fh9\" (UID: \"3094773b-d617-49d9-ac6a-435bd081ff21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feb5925-f7fh9" Oct 30 22:28:30.737908 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.737735 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcpq8\" (UniqueName: \"kubernetes.io/projected/dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26-kube-api-access-vcpq8\") pod \"maas-api-revoke-sub-33c6d78b-pdh26\" (UID: \"dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c6d78b-pdh26" Oct 30 22:28:30.737908 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.737883 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3094773b-d617-49d9-ac6a-435bd081ff21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0feb5925-f7fh9\" (UID: \"3094773b-d617-49d9-ac6a-435bd081ff21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feb5925-f7fh9" Oct 30 22:28:30.738072 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.737950 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3094773b-d617-49d9-ac6a-435bd081ff21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0feb5925-f7fh9\" (UID: \"3094773b-d617-49d9-ac6a-435bd081ff21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feb5925-f7fh9" Oct 30 22:28:30.738072 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.738027 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33c6d78b-pdh26\" (UID: \"dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c6d78b-pdh26" Oct 30 22:28:30.738187 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.738129 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33c6d78b-pdh26\" (UID: \"dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c6d78b-pdh26" Oct 30 22:28:30.738829 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.738804 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3094773b-d617-49d9-ac6a-435bd081ff21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0feb5925-f7fh9\" (UID: \"3094773b-d617-49d9-ac6a-435bd081ff21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feb5925-f7fh9" Oct 30 22:28:30.738829 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.738823 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33c6d78b-pdh26\" (UID: \"dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c6d78b-pdh26" Oct 30 22:28:30.740981 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.740963 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3094773b-d617-49d9-ac6a-435bd081ff21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0feb5925-f7fh9\" (UID: \"3094773b-d617-49d9-ac6a-435bd081ff21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feb5925-f7fh9" Oct 30 22:28:30.741059 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.741036 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33c6d78b-pdh26\" (UID: \"dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c6d78b-pdh26" Oct 30 22:28:30.796786 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.796750 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcpq8\" (UniqueName: \"kubernetes.io/projected/dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26-kube-api-access-vcpq8\") pod \"maas-api-revoke-sub-33c6d78b-pdh26\" (UID: \"dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c6d78b-pdh26" Oct 30 22:28:30.796976 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.796956 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhl6k\" (UniqueName: \"kubernetes.io/projected/3094773b-d617-49d9-ac6a-435bd081ff21-kube-api-access-dhl6k\") pod \"maas-api-revoke-sub-0feb5925-f7fh9\" (UID: \"3094773b-d617-49d9-ac6a-435bd081ff21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feb5925-f7fh9" Oct 30 22:28:30.882746 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.882712 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c6d78b-pdh26" Oct 30 22:28:30.890173 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:30.890146 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feb5925-f7fh9" Oct 30 22:28:31.262690 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:31.262590 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg" Oct 30 22:28:31.262690 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:31.262605 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg" event={"ID":"3b794876-ccb6-42a8-bdb9-7f35d3e97803","Type":"ContainerDied","Data":"341803e474439ad8f9db04d2eec7bb921c5c9150a3a424e753b9fd2b0f2a50a5"} Oct 30 22:28:31.262690 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:31.262663 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="341803e474439ad8f9db04d2eec7bb921c5c9150a3a424e753b9fd2b0f2a50a5" Oct 30 22:28:31.321323 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:31.320283 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33c6d78b-pdh26"] Oct 30 22:28:31.344474 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:28:31.344376 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc04d6d3_5b6b_4a09_a0c0_d8f41900ec26.slice/crio-c584802a12471aa7b54669f64b54a8bb06f57b45afd57f3bf62636ff92329c02 WatchSource:0}: Error finding container c584802a12471aa7b54669f64b54a8bb06f57b45afd57f3bf62636ff92329c02: Status 404 returned error can't find the container with id c584802a12471aa7b54669f64b54a8bb06f57b45afd57f3bf62636ff92329c02 Oct 30 22:28:31.356008 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:31.353288 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0feb5925-f7fh9"] Oct 30 22:28:31.357749 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:28:31.357718 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3094773b_d617_49d9_ac6a_435bd081ff21.slice/crio-1f43a126d397f81f0bb54066ac8a89c038559defb9dbef337bca602aa83b096b WatchSource:0}: Error finding container 1f43a126d397f81f0bb54066ac8a89c038559defb9dbef337bca602aa83b096b: Status 404 returned error can't find the container with id 1f43a126d397f81f0bb54066ac8a89c038559defb9dbef337bca602aa83b096b Oct 30 22:28:31.681753 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:31.681724 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:28:31.683027 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:31.683005 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:28:31.911722 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:31.911631 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs"] Oct 30 22:28:31.916392 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:31.916356 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs" Oct 30 22:28:31.973928 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:31.973876 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs"] Oct 30 22:28:32.068931 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:32.068893 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2wsj\" (UniqueName: \"kubernetes.io/projected/4f8aeb41-2899-4704-b20f-ba1ba69e1b74-kube-api-access-g2wsj\") pod \"maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs\" (UID: \"4f8aeb41-2899-4704-b20f-ba1ba69e1b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs" Oct 30 22:28:32.069151 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:32.069089 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f8aeb41-2899-4704-b20f-ba1ba69e1b74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs\" (UID: \"4f8aeb41-2899-4704-b20f-ba1ba69e1b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs" Oct 30 22:28:32.069229 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:32.069188 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f8aeb41-2899-4704-b20f-ba1ba69e1b74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs\" (UID: \"4f8aeb41-2899-4704-b20f-ba1ba69e1b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs" Oct 30 22:28:32.170490 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:32.170397 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f8aeb41-2899-4704-b20f-ba1ba69e1b74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs\" (UID: \"4f8aeb41-2899-4704-b20f-ba1ba69e1b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs" Oct 30 22:28:32.170654 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:32.170507 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2wsj\" (UniqueName: \"kubernetes.io/projected/4f8aeb41-2899-4704-b20f-ba1ba69e1b74-kube-api-access-g2wsj\") pod \"maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs\" (UID: \"4f8aeb41-2899-4704-b20f-ba1ba69e1b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs" Oct 30 22:28:32.170654 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:32.170614 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f8aeb41-2899-4704-b20f-ba1ba69e1b74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs\" (UID: \"4f8aeb41-2899-4704-b20f-ba1ba69e1b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs" Oct 30 22:28:32.171141 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:32.171118 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f8aeb41-2899-4704-b20f-ba1ba69e1b74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs\" (UID: \"4f8aeb41-2899-4704-b20f-ba1ba69e1b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs" Oct 30 22:28:32.173124 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:32.173094 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f8aeb41-2899-4704-b20f-ba1ba69e1b74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs\" (UID: \"4f8aeb41-2899-4704-b20f-ba1ba69e1b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs" Oct 30 22:28:32.234848 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:32.234807 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2wsj\" (UniqueName: \"kubernetes.io/projected/4f8aeb41-2899-4704-b20f-ba1ba69e1b74-kube-api-access-g2wsj\") pod \"maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs\" (UID: \"4f8aeb41-2899-4704-b20f-ba1ba69e1b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs" Oct 30 22:28:32.244562 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:32.244530 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs" Oct 30 22:28:32.273577 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:32.273529 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c6d78b-pdh26" event={"ID":"dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26","Type":"ContainerStarted","Data":"43549652394bbc3ee6bcbae1240962e1de7db88adb665819a4556ea9ccb0e512"} Oct 30 22:28:32.273761 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:32.273601 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c6d78b-pdh26" event={"ID":"dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26","Type":"ContainerStarted","Data":"c584802a12471aa7b54669f64b54a8bb06f57b45afd57f3bf62636ff92329c02"} Oct 30 22:28:32.275715 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:32.275679 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feb5925-f7fh9" event={"ID":"3094773b-d617-49d9-ac6a-435bd081ff21","Type":"ContainerStarted","Data":"ab25ce38de2e12c84f7fc71e76f106f4b1d462963bef1a047ce1dc7217dddf0b"} Oct 30 22:28:32.275866 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:32.275745 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feb5925-f7fh9" event={"ID":"3094773b-d617-49d9-ac6a-435bd081ff21","Type":"ContainerStarted","Data":"1f43a126d397f81f0bb54066ac8a89c038559defb9dbef337bca602aa83b096b"} Oct 30 22:28:32.389375 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:32.389241 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c6d78b-pdh26" podStartSLOduration=2.389225123 podStartE2EDuration="2.389225123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:32.387240085 +0000 UTC m=+2314.939416409" watchObservedRunningTime="2026-10-30 22:28:32.389225123 +0000 UTC m=+2314.941401426" Oct 30 22:28:32.501516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:32.501451 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feb5925-f7fh9" podStartSLOduration=2.501436794 podStartE2EDuration="2.501436794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:32.496970782 +0000 UTC m=+2315.049147081" watchObservedRunningTime="2026-10-30 22:28:32.501436794 +0000 UTC m=+2315.053613093" Oct 30 22:28:32.622561 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:32.622534 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs"] Oct 30 22:28:32.633493 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:28:32.633462 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8aeb41_2899_4704_b20f_ba1ba69e1b74.slice/crio-876b7d5ed4d69cad2469b0257e91e85c3ad87295cb6b15f18ead5692bfc9ec75 WatchSource:0}: Error finding container 876b7d5ed4d69cad2469b0257e91e85c3ad87295cb6b15f18ead5692bfc9ec75: Status 404 returned error can't find the container with id 876b7d5ed4d69cad2469b0257e91e85c3ad87295cb6b15f18ead5692bfc9ec75 Oct 30 22:28:33.207826 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:33.207729 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz"] Oct 30 22:28:33.284324 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:33.284277 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs" event={"ID":"4f8aeb41-2899-4704-b20f-ba1ba69e1b74","Type":"ContainerStarted","Data":"9ae75b13eb448049ab4337698d182de7d74239ecf4324004293d6b07747c5be8"} Oct 30 22:28:33.284537 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:33.284348 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs" event={"ID":"4f8aeb41-2899-4704-b20f-ba1ba69e1b74","Type":"ContainerStarted","Data":"876b7d5ed4d69cad2469b0257e91e85c3ad87295cb6b15f18ead5692bfc9ec75"} Oct 30 22:28:33.286077 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:33.286046 6006 generic.go:356] "Generic (PLEG): container finished" podID="dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26" containerID="43549652394bbc3ee6bcbae1240962e1de7db88adb665819a4556ea9ccb0e512" exitCode=0 Oct 30 22:28:33.286203 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:33.286138 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c6d78b-pdh26" event={"ID":"dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26","Type":"ContainerDied","Data":"43549652394bbc3ee6bcbae1240962e1de7db88adb665819a4556ea9ccb0e512"} Oct 30 22:28:33.287904 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:33.287882 6006 generic.go:356] "Generic (PLEG): container finished" podID="3094773b-d617-49d9-ac6a-435bd081ff21" containerID="ab25ce38de2e12c84f7fc71e76f106f4b1d462963bef1a047ce1dc7217dddf0b" exitCode=0 Oct 30 22:28:33.288027 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:33.287972 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feb5925-f7fh9" event={"ID":"3094773b-d617-49d9-ac6a-435bd081ff21","Type":"ContainerDied","Data":"ab25ce38de2e12c84f7fc71e76f106f4b1d462963bef1a047ce1dc7217dddf0b"} Oct 30 22:28:33.326894 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:33.326857 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp"] Oct 30 22:28:33.713919 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:33.713856 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs" podStartSLOduration=2.713841691 podStartE2EDuration="2.713841691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:33.710037041 +0000 UTC m=+2316.262213340" watchObservedRunningTime="2026-10-30 22:28:33.713841691 +0000 UTC m=+2316.266017982" Oct 30 22:28:34.301195 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:34.301154 6006 generic.go:356] "Generic (PLEG): container finished" podID="4f8aeb41-2899-4704-b20f-ba1ba69e1b74" containerID="9ae75b13eb448049ab4337698d182de7d74239ecf4324004293d6b07747c5be8" exitCode=0 Oct 30 22:28:34.301904 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:34.301589 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs" event={"ID":"4f8aeb41-2899-4704-b20f-ba1ba69e1b74","Type":"ContainerDied","Data":"9ae75b13eb448049ab4337698d182de7d74239ecf4324004293d6b07747c5be8"} Oct 30 22:28:34.550582 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:34.550552 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feb5925-f7fh9" Oct 30 22:28:34.614310 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:34.614229 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3094773b-d617-49d9-ac6a-435bd081ff21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3094773b-d617-49d9-ac6a-435bd081ff21-maas-api-openshift-ca\") pod \"3094773b-d617-49d9-ac6a-435bd081ff21\" (UID: \"3094773b-d617-49d9-ac6a-435bd081ff21\") " Oct 30 22:28:34.614543 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:34.614519 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3094773b-d617-49d9-ac6a-435bd081ff21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3094773b-d617-49d9-ac6a-435bd081ff21-maas-api-xks-ca\") pod \"3094773b-d617-49d9-ac6a-435bd081ff21\" (UID: \"3094773b-d617-49d9-ac6a-435bd081ff21\") " Oct 30 22:28:34.614643 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:34.614601 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3094773b-d617-49d9-ac6a-435bd081ff21-kube-api-access-dhl6k\" (UniqueName: \"kubernetes.io/projected/3094773b-d617-49d9-ac6a-435bd081ff21-kube-api-access-dhl6k\") pod \"3094773b-d617-49d9-ac6a-435bd081ff21\" (UID: \"3094773b-d617-49d9-ac6a-435bd081ff21\") " Oct 30 22:28:34.614697 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:34.614633 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3094773b-d617-49d9-ac6a-435bd081ff21-maas-api-openshift-ca" pod "3094773b-d617-49d9-ac6a-435bd081ff21" (UID: "3094773b-d617-49d9-ac6a-435bd081ff21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:34.615207 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:34.615178 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3094773b-d617-49d9-ac6a-435bd081ff21-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:34.617371 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:34.617340 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3094773b-d617-49d9-ac6a-435bd081ff21-maas-api-xks-ca" pod "3094773b-d617-49d9-ac6a-435bd081ff21" (UID: "3094773b-d617-49d9-ac6a-435bd081ff21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:34.617498 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:34.617410 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3094773b-d617-49d9-ac6a-435bd081ff21-kube-api-access-dhl6k" pod "3094773b-d617-49d9-ac6a-435bd081ff21" (UID: "3094773b-d617-49d9-ac6a-435bd081ff21"). InnerVolumeSpecName "kube-api-access-dhl6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:34.620470 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:34.620449 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c6d78b-pdh26" Oct 30 22:28:34.736292 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:34.719992 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26-kube-api-access-vcpq8\" (UniqueName: \"kubernetes.io/projected/dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26-kube-api-access-vcpq8\") pod \"dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26\" (UID: \"dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26\") " Oct 30 22:28:34.736292 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:34.720155 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26-maas-api-openshift-ca\") pod \"dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26\" (UID: \"dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26\") " Oct 30 22:28:34.736292 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:34.720189 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26-maas-api-xks-ca\") pod \"dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26\" (UID: \"dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26\") " Oct 30 22:28:34.736292 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:34.721154 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3094773b-d617-49d9-ac6a-435bd081ff21-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:34.736292 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:34.721182 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhl6k\" (UniqueName: \"kubernetes.io/projected/3094773b-d617-49d9-ac6a-435bd081ff21-kube-api-access-dhl6k\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:34.736292 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:34.726592 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26-maas-api-openshift-ca" pod "dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26" (UID: "dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:34.762849 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:34.762789 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26-maas-api-xks-ca" pod "dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26" (UID: "dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:34.780812 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:34.780759 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26-kube-api-access-vcpq8" pod "dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26" (UID: "dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26"). InnerVolumeSpecName "kube-api-access-vcpq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:34.830704 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:34.823673 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcpq8\" (UniqueName: \"kubernetes.io/projected/dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26-kube-api-access-vcpq8\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:34.830704 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:34.823722 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:34.830704 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:34.823743 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:35.313034 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:35.312995 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c6d78b-pdh26" Oct 30 22:28:35.313514 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:35.313002 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c6d78b-pdh26" event={"ID":"dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26","Type":"ContainerDied","Data":"c584802a12471aa7b54669f64b54a8bb06f57b45afd57f3bf62636ff92329c02"} Oct 30 22:28:35.313514 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:35.313174 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c584802a12471aa7b54669f64b54a8bb06f57b45afd57f3bf62636ff92329c02" Oct 30 22:28:35.314886 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:35.314865 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feb5925-f7fh9" Oct 30 22:28:35.315015 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:35.314882 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0feb5925-f7fh9" event={"ID":"3094773b-d617-49d9-ac6a-435bd081ff21","Type":"ContainerDied","Data":"1f43a126d397f81f0bb54066ac8a89c038559defb9dbef337bca602aa83b096b"} Oct 30 22:28:35.315015 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:35.314929 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f43a126d397f81f0bb54066ac8a89c038559defb9dbef337bca602aa83b096b" Oct 30 22:28:35.509033 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:35.509005 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs" Oct 30 22:28:35.559387 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:35.559342 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f8aeb41-2899-4704-b20f-ba1ba69e1b74-kube-api-access-g2wsj\" (UniqueName: \"kubernetes.io/projected/4f8aeb41-2899-4704-b20f-ba1ba69e1b74-kube-api-access-g2wsj\") pod \"4f8aeb41-2899-4704-b20f-ba1ba69e1b74\" (UID: \"4f8aeb41-2899-4704-b20f-ba1ba69e1b74\") " Oct 30 22:28:35.559599 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:35.559492 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f8aeb41-2899-4704-b20f-ba1ba69e1b74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f8aeb41-2899-4704-b20f-ba1ba69e1b74-maas-api-xks-ca\") pod \"4f8aeb41-2899-4704-b20f-ba1ba69e1b74\" (UID: \"4f8aeb41-2899-4704-b20f-ba1ba69e1b74\") " Oct 30 22:28:35.559599 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:35.559529 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f8aeb41-2899-4704-b20f-ba1ba69e1b74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f8aeb41-2899-4704-b20f-ba1ba69e1b74-maas-api-openshift-ca\") pod \"4f8aeb41-2899-4704-b20f-ba1ba69e1b74\" (UID: \"4f8aeb41-2899-4704-b20f-ba1ba69e1b74\") " Oct 30 22:28:35.560239 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:35.560212 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f8aeb41-2899-4704-b20f-ba1ba69e1b74-maas-api-openshift-ca" pod "4f8aeb41-2899-4704-b20f-ba1ba69e1b74" (UID: "4f8aeb41-2899-4704-b20f-ba1ba69e1b74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:35.561660 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:35.561629 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8aeb41-2899-4704-b20f-ba1ba69e1b74-kube-api-access-g2wsj" pod "4f8aeb41-2899-4704-b20f-ba1ba69e1b74" (UID: "4f8aeb41-2899-4704-b20f-ba1ba69e1b74"). InnerVolumeSpecName "kube-api-access-g2wsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:35.562560 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:35.562529 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f8aeb41-2899-4704-b20f-ba1ba69e1b74-maas-api-xks-ca" pod "4f8aeb41-2899-4704-b20f-ba1ba69e1b74" (UID: "4f8aeb41-2899-4704-b20f-ba1ba69e1b74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:35.661497 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:35.661434 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f8aeb41-2899-4704-b20f-ba1ba69e1b74-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:35.661497 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:35.661501 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f8aeb41-2899-4704-b20f-ba1ba69e1b74-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:35.661767 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:35.661519 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2wsj\" (UniqueName: \"kubernetes.io/projected/4f8aeb41-2899-4704-b20f-ba1ba69e1b74-kube-api-access-g2wsj\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:36.325293 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:36.325214 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs" Oct 30 22:28:36.325293 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:36.325233 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs" event={"ID":"4f8aeb41-2899-4704-b20f-ba1ba69e1b74","Type":"ContainerDied","Data":"876b7d5ed4d69cad2469b0257e91e85c3ad87295cb6b15f18ead5692bfc9ec75"} Oct 30 22:28:36.325742 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:36.325329 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="876b7d5ed4d69cad2469b0257e91e85c3ad87295cb6b15f18ead5692bfc9ec75" Oct 30 22:28:36.350553 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:36.350512 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp"] Oct 30 22:28:37.327124 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:37.327085 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq"] Oct 30 22:28:38.208325 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:38.208279 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz" podUID="695df2fa-fb27-4a50-995e-acb44e355440" containerName="payload-processing" containerID="cri-o://acc438b39afb01ec9ca8cf95158f5add3753a4e25fcd418e94de976e1a47676c" gracePeriod=25 Oct 30 22:28:38.328241 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:38.328195 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp" podUID="d0b10d96-ee84-4a97-b5de-20e08008815d" containerName="payload-pre-processing" containerID="cri-o://6f38731e9a9354379ba0bc76fb7562d3d1283ccdafe79923a502af1ff01cfc4e" gracePeriod=25 Oct 30 22:28:38.348536 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:38.348501 6006 generic.go:356] "Generic (PLEG): container finished" podID="695df2fa-fb27-4a50-995e-acb44e355440" containerID="acc438b39afb01ec9ca8cf95158f5add3753a4e25fcd418e94de976e1a47676c" exitCode=0 Oct 30 22:28:38.348689 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:38.348539 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz" event={"ID":"695df2fa-fb27-4a50-995e-acb44e355440","Type":"ContainerDied","Data":"acc438b39afb01ec9ca8cf95158f5add3753a4e25fcd418e94de976e1a47676c"} Oct 30 22:28:38.516183 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:38.515574 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz" Oct 30 22:28:38.599307 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:38.599266 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/695df2fa-fb27-4a50-995e-acb44e355440-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695df2fa-fb27-4a50-995e-acb44e355440-plugins-config-volume\") pod \"695df2fa-fb27-4a50-995e-acb44e355440\" (UID: \"695df2fa-fb27-4a50-995e-acb44e355440\") " Oct 30 22:28:38.599522 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:38.599500 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/695df2fa-fb27-4a50-995e-acb44e355440-kube-api-access-5rgzn\" (UniqueName: \"kubernetes.io/projected/695df2fa-fb27-4a50-995e-acb44e355440-kube-api-access-5rgzn\") pod \"695df2fa-fb27-4a50-995e-acb44e355440\" (UID: \"695df2fa-fb27-4a50-995e-acb44e355440\") " Oct 30 22:28:38.600596 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:38.600568 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695df2fa-fb27-4a50-995e-acb44e355440-plugins-config-volume" pod "695df2fa-fb27-4a50-995e-acb44e355440" (UID: "695df2fa-fb27-4a50-995e-acb44e355440"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:38.609803 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:38.603480 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695df2fa-fb27-4a50-995e-acb44e355440-kube-api-access-5rgzn" pod "695df2fa-fb27-4a50-995e-acb44e355440" (UID: "695df2fa-fb27-4a50-995e-acb44e355440"). InnerVolumeSpecName "kube-api-access-5rgzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:38.645197 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:38.645161 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp" Oct 30 22:28:38.714197 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:38.711592 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0b10d96-ee84-4a97-b5de-20e08008815d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0b10d96-ee84-4a97-b5de-20e08008815d-plugins-config-volume\") pod \"d0b10d96-ee84-4a97-b5de-20e08008815d\" (UID: \"d0b10d96-ee84-4a97-b5de-20e08008815d\") " Oct 30 22:28:38.714197 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:38.711727 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0b10d96-ee84-4a97-b5de-20e08008815d-kube-api-access-jhssd\" (UniqueName: \"kubernetes.io/projected/d0b10d96-ee84-4a97-b5de-20e08008815d-kube-api-access-jhssd\") pod \"d0b10d96-ee84-4a97-b5de-20e08008815d\" (UID: \"d0b10d96-ee84-4a97-b5de-20e08008815d\") " Oct 30 22:28:38.752363 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:38.740511 6006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695df2fa-fb27-4a50-995e-acb44e355440-plugins-config-volume\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:38.752363 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:38.740554 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rgzn\" (UniqueName: \"kubernetes.io/projected/695df2fa-fb27-4a50-995e-acb44e355440-kube-api-access-5rgzn\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:38.768272 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:38.767710 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0b10d96-ee84-4a97-b5de-20e08008815d-plugins-config-volume" pod "d0b10d96-ee84-4a97-b5de-20e08008815d" (UID: "d0b10d96-ee84-4a97-b5de-20e08008815d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:38.787524 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:38.787451 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0b10d96-ee84-4a97-b5de-20e08008815d-kube-api-access-jhssd" pod "d0b10d96-ee84-4a97-b5de-20e08008815d" (UID: "d0b10d96-ee84-4a97-b5de-20e08008815d"). InnerVolumeSpecName "kube-api-access-jhssd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:38.848171 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:38.848083 6006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0b10d96-ee84-4a97-b5de-20e08008815d-plugins-config-volume\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:38.848171 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:38.848128 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhssd\" (UniqueName: \"kubernetes.io/projected/d0b10d96-ee84-4a97-b5de-20e08008815d-kube-api-access-jhssd\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:39.367945 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:39.366725 6006 generic.go:356] "Generic (PLEG): container finished" podID="d0b10d96-ee84-4a97-b5de-20e08008815d" containerID="6f38731e9a9354379ba0bc76fb7562d3d1283ccdafe79923a502af1ff01cfc4e" exitCode=0 Oct 30 22:28:39.367945 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:39.367007 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp" Oct 30 22:28:39.368864 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:39.368818 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp" event={"ID":"d0b10d96-ee84-4a97-b5de-20e08008815d","Type":"ContainerDied","Data":"6f38731e9a9354379ba0bc76fb7562d3d1283ccdafe79923a502af1ff01cfc4e"} Oct 30 22:28:39.369097 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:39.369074 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp" event={"ID":"d0b10d96-ee84-4a97-b5de-20e08008815d","Type":"ContainerDied","Data":"dc5fd9192392cf4d008eab1643b0e0bc9a80e0ea648e3ddf6d4c5db473c472f5"} Oct 30 22:28:39.369202 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:39.369116 6006 scope.go:122] "RemoveContainer" containerID="6f38731e9a9354379ba0bc76fb7562d3d1283ccdafe79923a502af1ff01cfc4e" Oct 30 22:28:39.370681 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:39.370378 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz" event={"ID":"695df2fa-fb27-4a50-995e-acb44e355440","Type":"ContainerDied","Data":"c7b84de1dc000bd843f5db69fc740936148c8a2363cb24287d1b77a3aef6da48"} Oct 30 22:28:39.370681 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:39.370422 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz" Oct 30 22:28:39.382754 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:39.382734 6006 scope.go:122] "RemoveContainer" containerID="6f38731e9a9354379ba0bc76fb7562d3d1283ccdafe79923a502af1ff01cfc4e" Oct 30 22:28:39.383077 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:28:39.383058 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f38731e9a9354379ba0bc76fb7562d3d1283ccdafe79923a502af1ff01cfc4e\": container with ID starting with 6f38731e9a9354379ba0bc76fb7562d3d1283ccdafe79923a502af1ff01cfc4e not found: ID does not exist" containerID="6f38731e9a9354379ba0bc76fb7562d3d1283ccdafe79923a502af1ff01cfc4e" Oct 30 22:28:39.383160 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:39.383088 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f38731e9a9354379ba0bc76fb7562d3d1283ccdafe79923a502af1ff01cfc4e"} err="failed to get container status \"6f38731e9a9354379ba0bc76fb7562d3d1283ccdafe79923a502af1ff01cfc4e\": rpc error: code = NotFound desc = could not find container \"6f38731e9a9354379ba0bc76fb7562d3d1283ccdafe79923a502af1ff01cfc4e\": container with ID starting with 6f38731e9a9354379ba0bc76fb7562d3d1283ccdafe79923a502af1ff01cfc4e not found: ID does not exist" Oct 30 22:28:39.383160 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:39.383111 6006 scope.go:122] "RemoveContainer" containerID="acc438b39afb01ec9ca8cf95158f5add3753a4e25fcd418e94de976e1a47676c" Oct 30 22:28:39.530650 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:39.530566 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp"] Oct 30 22:28:39.578090 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:39.578050 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d7cdcc-7754fd7967-m2vbp"] Oct 30 22:28:39.605976 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:39.605936 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0b10d96-ee84-4a97-b5de-20e08008815d" path="/var/lib/kubelet/pods/d0b10d96-ee84-4a97-b5de-20e08008815d/volumes" Oct 30 22:28:39.679678 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:39.679645 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz"] Oct 30 22:28:39.684595 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:39.684561 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d7cdcc-69d9954cd5-sftsz"] Oct 30 22:28:41.605831 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:41.605782 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="695df2fa-fb27-4a50-995e-acb44e355440" path="/var/lib/kubelet/pods/695df2fa-fb27-4a50-995e-acb44e355440/volumes" Oct 30 22:28:42.331997 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:42.331432 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq" podUID="5ed80968-c4fc-4ba3-8382-97c1d865b5ac" containerName="payload-processing" containerID="cri-o://7fcee6c69caf808baea6fc7b46dee2992cc5f9512a34e4522bed263adffc7ffc" gracePeriod=25 Oct 30 22:28:42.650727 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:42.650398 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq" Oct 30 22:28:42.758807 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:42.758762 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ed80968-c4fc-4ba3-8382-97c1d865b5ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed80968-c4fc-4ba3-8382-97c1d865b5ac-plugins-config-volume\") pod \"5ed80968-c4fc-4ba3-8382-97c1d865b5ac\" (UID: \"5ed80968-c4fc-4ba3-8382-97c1d865b5ac\") " Oct 30 22:28:42.759020 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:42.758844 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ed80968-c4fc-4ba3-8382-97c1d865b5ac-kube-api-access-gll26\" (UniqueName: \"kubernetes.io/projected/5ed80968-c4fc-4ba3-8382-97c1d865b5ac-kube-api-access-gll26\") pod \"5ed80968-c4fc-4ba3-8382-97c1d865b5ac\" (UID: \"5ed80968-c4fc-4ba3-8382-97c1d865b5ac\") " Oct 30 22:28:42.759203 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:42.759166 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ed80968-c4fc-4ba3-8382-97c1d865b5ac-plugins-config-volume" pod "5ed80968-c4fc-4ba3-8382-97c1d865b5ac" (UID: "5ed80968-c4fc-4ba3-8382-97c1d865b5ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:42.759380 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:42.759363 6006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed80968-c4fc-4ba3-8382-97c1d865b5ac-plugins-config-volume\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:42.762747 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:42.761327 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed80968-c4fc-4ba3-8382-97c1d865b5ac-kube-api-access-gll26" pod "5ed80968-c4fc-4ba3-8382-97c1d865b5ac" (UID: "5ed80968-c4fc-4ba3-8382-97c1d865b5ac"). InnerVolumeSpecName "kube-api-access-gll26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:42.837774 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:42.837723 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf"] Oct 30 22:28:42.842580 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:42.842540 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7cdcc-8qvnf"] Oct 30 22:28:42.860342 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:42.860199 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gll26\" (UniqueName: \"kubernetes.io/projected/5ed80968-c4fc-4ba3-8382-97c1d865b5ac-kube-api-access-gll26\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:43.423830 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:43.423789 6006 generic.go:356] "Generic (PLEG): container finished" podID="5ed80968-c4fc-4ba3-8382-97c1d865b5ac" containerID="7fcee6c69caf808baea6fc7b46dee2992cc5f9512a34e4522bed263adffc7ffc" exitCode=0 Oct 30 22:28:43.424059 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:43.424018 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq" event={"ID":"5ed80968-c4fc-4ba3-8382-97c1d865b5ac","Type":"ContainerDied","Data":"7fcee6c69caf808baea6fc7b46dee2992cc5f9512a34e4522bed263adffc7ffc"} Oct 30 22:28:43.424229 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:43.424122 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq" event={"ID":"5ed80968-c4fc-4ba3-8382-97c1d865b5ac","Type":"ContainerDied","Data":"376e6786a798549d5cf0117a27475dbcdcda26ea9e3d33c1fbc5ea35a726099e"} Oct 30 22:28:43.424229 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:43.424182 6006 scope.go:122] "RemoveContainer" containerID="7fcee6c69caf808baea6fc7b46dee2992cc5f9512a34e4522bed263adffc7ffc" Oct 30 22:28:43.424413 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:43.424303 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq" Oct 30 22:28:43.439618 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:43.439588 6006 scope.go:122] "RemoveContainer" containerID="7fcee6c69caf808baea6fc7b46dee2992cc5f9512a34e4522bed263adffc7ffc" Oct 30 22:28:43.440010 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:28:43.439975 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fcee6c69caf808baea6fc7b46dee2992cc5f9512a34e4522bed263adffc7ffc\": container with ID starting with 7fcee6c69caf808baea6fc7b46dee2992cc5f9512a34e4522bed263adffc7ffc not found: ID does not exist" containerID="7fcee6c69caf808baea6fc7b46dee2992cc5f9512a34e4522bed263adffc7ffc" Oct 30 22:28:43.440141 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:43.440021 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fcee6c69caf808baea6fc7b46dee2992cc5f9512a34e4522bed263adffc7ffc"} err="failed to get container status \"7fcee6c69caf808baea6fc7b46dee2992cc5f9512a34e4522bed263adffc7ffc\": rpc error: code = NotFound desc = could not find container \"7fcee6c69caf808baea6fc7b46dee2992cc5f9512a34e4522bed263adffc7ffc\": container with ID starting with 7fcee6c69caf808baea6fc7b46dee2992cc5f9512a34e4522bed263adffc7ffc not found: ID does not exist" Oct 30 22:28:43.594210 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:43.594171 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq"] Oct 30 22:28:43.605436 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:43.605402 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abbf73e3-26bb-4927-91b7-c0ece65c8fbb" path="/var/lib/kubelet/pods/abbf73e3-26bb-4927-91b7-c0ece65c8fbb/volumes" Oct 30 22:28:43.605777 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:43.605761 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0b6a3f-86869cb7ff-5kbxq"] Oct 30 22:28:44.088968 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:44.088919 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2"] Oct 30 22:28:44.206307 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:44.202926 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg"] Oct 30 22:28:44.248682 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:44.248641 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b6a3f-5k2vg"] Oct 30 22:28:45.053736 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:45.053704 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f93eab74-nrqt7"] Oct 30 22:28:45.056836 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:45.056805 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f93eab74-nrqt7"] Oct 30 22:28:45.627277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:45.626994 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b794876-ccb6-42a8-bdb9-7f35d3e97803" path="/var/lib/kubelet/pods/3b794876-ccb6-42a8-bdb9-7f35d3e97803/volumes" Oct 30 22:28:45.627818 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:45.627713 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b33669b-cc84-48a7-a786-d7689ca52e5c" path="/var/lib/kubelet/pods/4b33669b-cc84-48a7-a786-d7689ca52e5c/volumes" Oct 30 22:28:45.637279 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:45.628275 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ed80968-c4fc-4ba3-8382-97c1d865b5ac" path="/var/lib/kubelet/pods/5ed80968-c4fc-4ba3-8382-97c1d865b5ac/volumes" Oct 30 22:28:49.090324 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:49.090235 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2" podUID="9a510562-a7e8-4abe-963e-3ddd4ff88162" containerName="payload-processing" containerID="cri-o://f435bc68dd004f99ef69ad5739aca3634b4343eff946ebef1231e5ecffa6c2d9" gracePeriod=25 Oct 30 22:28:49.719910 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:49.719866 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2" podUID="9a510562-a7e8-4abe-963e-3ddd4ff88162" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:28:50.138828 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:50.138788 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp"] Oct 30 22:28:50.139381 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:50.139043 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" podUID="f9c79304-5ef5-4fc9-ba42-facc6c16f2b0" containerName="istio-proxy" containerID="cri-o://f8c67a25f196c2cfb0fc6182c9ad8f31bf9995776e1d2f5226c0a7662cc7973c" gracePeriod=30 Oct 30 22:28:51.372505 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:51.372443 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" containerID="cri-o://1e4b39dcd309429e18613b9593aaa5b1372fca9c9cf0444663d42d3a7c32e203" gracePeriod=45 Oct 30 22:28:51.490381 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:51.490347 6006 generic.go:356] "Generic (PLEG): container finished" podID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerID="1e4b39dcd309429e18613b9593aaa5b1372fca9c9cf0444663d42d3a7c32e203" exitCode=0 Oct 30 22:28:51.490508 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:51.490414 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" event={"ID":"636d5b64-a287-4a2b-bee0-db3b0184b0f1","Type":"ContainerDied","Data":"1e4b39dcd309429e18613b9593aaa5b1372fca9c9cf0444663d42d3a7c32e203"} Oct 30 22:28:51.668886 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:51.668859 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:28:51.751680 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:51.751644 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/636d5b64-a287-4a2b-bee0-db3b0184b0f1-tls-certs\" (UniqueName: \"kubernetes.io/secret/636d5b64-a287-4a2b-bee0-db3b0184b0f1-tls-certs\") pod \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\" (UID: \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\") " Oct 30 22:28:51.751888 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:51.751727 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-tmp-dir\") pod \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\" (UID: \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\") " Oct 30 22:28:51.751888 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:51.751803 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636d5b64-a287-4a2b-bee0-db3b0184b0f1-kube-api-access-7lpgw\" (UniqueName: \"kubernetes.io/projected/636d5b64-a287-4a2b-bee0-db3b0184b0f1-kube-api-access-7lpgw\") pod \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\" (UID: \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\") " Oct 30 22:28:51.751888 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:51.751829 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-dshm\") pod \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\" (UID: \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\") " Oct 30 22:28:51.751888 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:51.751854 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-model-cache\") pod \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\" (UID: \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\") " Oct 30 22:28:51.752127 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:51.751893 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-home\" (UniqueName: \"kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-home\") pod \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\" (UID: \"636d5b64-a287-4a2b-bee0-db3b0184b0f1\") " Oct 30 22:28:51.752127 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:51.752054 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-tmp-dir" pod "636d5b64-a287-4a2b-bee0-db3b0184b0f1" (UID: "636d5b64-a287-4a2b-bee0-db3b0184b0f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:51.752280 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:51.752220 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-home" pod "636d5b64-a287-4a2b-bee0-db3b0184b0f1" (UID: "636d5b64-a287-4a2b-bee0-db3b0184b0f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:51.752406 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:51.752382 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-model-cache" pod "636d5b64-a287-4a2b-bee0-db3b0184b0f1" (UID: "636d5b64-a287-4a2b-bee0-db3b0184b0f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:51.752516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:51.752491 6006 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-home\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:51.752516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:51.752516 6006 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-tmp-dir\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:51.754148 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:51.754124 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/636d5b64-a287-4a2b-bee0-db3b0184b0f1-tls-certs" pod "636d5b64-a287-4a2b-bee0-db3b0184b0f1" (UID: "636d5b64-a287-4a2b-bee0-db3b0184b0f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:51.754226 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:51.754133 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-dshm" pod "636d5b64-a287-4a2b-bee0-db3b0184b0f1" (UID: "636d5b64-a287-4a2b-bee0-db3b0184b0f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:51.754226 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:51.754158 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636d5b64-a287-4a2b-bee0-db3b0184b0f1-kube-api-access-7lpgw" pod "636d5b64-a287-4a2b-bee0-db3b0184b0f1" (UID: "636d5b64-a287-4a2b-bee0-db3b0184b0f1"). InnerVolumeSpecName "kube-api-access-7lpgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:51.854123 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:51.854091 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lpgw\" (UniqueName: \"kubernetes.io/projected/636d5b64-a287-4a2b-bee0-db3b0184b0f1-kube-api-access-7lpgw\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:51.854123 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:51.854125 6006 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-dshm\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:51.854123 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:51.854139 6006 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/636d5b64-a287-4a2b-bee0-db3b0184b0f1-model-cache\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:51.854408 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:51.854153 6006 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/636d5b64-a287-4a2b-bee0-db3b0184b0f1-tls-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:52.081893 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.081848 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx"] Oct 30 22:28:52.082126 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.082098 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" podUID="0ec694be-68a9-4227-96ac-cf7daac3af60" containerName="istio-proxy" containerID="cri-o://09efbed3d73532097e2c8cc48b361c1b72d96e02302e6a9927708c8f929606e5" gracePeriod=30 Oct 30 22:28:52.499724 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.499615 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" event={"ID":"636d5b64-a287-4a2b-bee0-db3b0184b0f1","Type":"ContainerDied","Data":"7ff4cfe3a188d4e125e4a39436d34a8ac21ec7701dcc71eae7c43ed69c4be227"} Oct 30 22:28:52.499724 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.499687 6006 scope.go:122] "RemoveContainer" containerID="1e4b39dcd309429e18613b9593aaa5b1372fca9c9cf0444663d42d3a7c32e203" Oct 30 22:28:52.501227 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.501047 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp" Oct 30 22:28:52.502855 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.502832 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5"] Oct 30 22:28:52.503819 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.503796 6006 cpu_manager.go:414] "containerMap: removing container" podUID="3094773b-d617-49d9-ac6a-435bd081ff21" containerName="revoke-keys" Oct 30 22:28:52.503911 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.503819 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3094773b-d617-49d9-ac6a-435bd081ff21" containerName="revoke-keys" Oct 30 22:28:52.503911 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.503846 6006 cpu_manager.go:414] "containerMap: removing container" podUID="695df2fa-fb27-4a50-995e-acb44e355440" containerName="payload-processing" Oct 30 22:28:52.503911 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.503855 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="695df2fa-fb27-4a50-995e-acb44e355440" containerName="payload-processing" Oct 30 22:28:52.503911 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.503868 6006 cpu_manager.go:414] "containerMap: removing container" podUID="4f8aeb41-2899-4704-b20f-ba1ba69e1b74" containerName="revoke-keys" Oct 30 22:28:52.503911 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.503875 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f8aeb41-2899-4704-b20f-ba1ba69e1b74" containerName="revoke-keys" Oct 30 22:28:52.504078 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.503916 6006 cpu_manager.go:414] "containerMap: removing container" podUID="5ed80968-c4fc-4ba3-8382-97c1d865b5ac" containerName="payload-processing" Oct 30 22:28:52.504078 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.503924 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ed80968-c4fc-4ba3-8382-97c1d865b5ac" containerName="payload-processing" Oct 30 22:28:52.504078 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.503936 6006 cpu_manager.go:414] "containerMap: removing container" podUID="dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26" containerName="revoke-keys" Oct 30 22:28:52.504078 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.503943 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26" containerName="revoke-keys" Oct 30 22:28:52.504078 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.503957 6006 cpu_manager.go:414] "containerMap: removing container" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" Oct 30 22:28:52.504078 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.503965 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" Oct 30 22:28:52.504078 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.504001 6006 cpu_manager.go:414] "containerMap: removing container" podUID="d0b10d96-ee84-4a97-b5de-20e08008815d" containerName="payload-pre-processing" Oct 30 22:28:52.504078 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.504009 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0b10d96-ee84-4a97-b5de-20e08008815d" containerName="payload-pre-processing" Oct 30 22:28:52.504381 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.504179 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ed80968-c4fc-4ba3-8382-97c1d865b5ac" containerName="payload-processing" Oct 30 22:28:52.504381 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.504193 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26" containerName="revoke-keys" Oct 30 22:28:52.504381 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.504206 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" containerName="main" Oct 30 22:28:52.504381 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.504221 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0b10d96-ee84-4a97-b5de-20e08008815d" containerName="payload-pre-processing" Oct 30 22:28:52.504381 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.504238 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="3094773b-d617-49d9-ac6a-435bd081ff21" containerName="revoke-keys" Oct 30 22:28:52.504381 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.504263 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="695df2fa-fb27-4a50-995e-acb44e355440" containerName="payload-processing" Oct 30 22:28:52.504381 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.504273 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f8aeb41-2899-4704-b20f-ba1ba69e1b74" containerName="revoke-keys" Oct 30 22:28:52.514192 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.514162 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5" Oct 30 22:28:52.572847 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.572809 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5\" (UID: \"f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5" Oct 30 22:28:52.573064 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.572988 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfnph\" (UniqueName: \"kubernetes.io/projected/f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4-kube-api-access-dfnph\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5\" (UID: \"f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5" Oct 30 22:28:52.573064 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.573032 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5\" (UID: \"f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5" Oct 30 22:28:52.598620 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.598576 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:28:52.598822 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.598676 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:28:52.606738 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.606700 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5"] Oct 30 22:28:52.660275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.660228 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp"] Oct 30 22:28:52.665145 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.665116 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-65e59f/distinct-w4-65e59f-kserve-58947cc74f-r2tsp"] Oct 30 22:28:52.674393 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.674357 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfnph\" (UniqueName: \"kubernetes.io/projected/f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4-kube-api-access-dfnph\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5\" (UID: \"f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5" Oct 30 22:28:52.674557 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.674411 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5\" (UID: \"f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5" Oct 30 22:28:52.674624 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.674568 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5\" (UID: \"f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5" Oct 30 22:28:52.675115 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.675092 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5\" (UID: \"f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5" Oct 30 22:28:52.676894 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.676873 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5\" (UID: \"f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5" Oct 30 22:28:52.736613 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.736574 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfnph\" (UniqueName: \"kubernetes.io/projected/f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4-kube-api-access-dfnph\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5\" (UID: \"f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5" Oct 30 22:28:52.835422 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:52.835385 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5" Oct 30 22:28:53.218641 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:53.218605 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5"] Oct 30 22:28:53.219763 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:28:53.219732 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf49457d4_4c5f_41f2_b8d1_bbdb4357a6d4.slice/crio-75392af1af5992bae2a0181c046ca942fa1d8acee0da5d88c8491adaae112b40 WatchSource:0}: Error finding container 75392af1af5992bae2a0181c046ca942fa1d8acee0da5d88c8491adaae112b40: Status 404 returned error can't find the container with id 75392af1af5992bae2a0181c046ca942fa1d8acee0da5d88c8491adaae112b40 Oct 30 22:28:53.511484 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:53.511393 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5" event={"ID":"f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4","Type":"ContainerStarted","Data":"75392af1af5992bae2a0181c046ca942fa1d8acee0da5d88c8491adaae112b40"} Oct 30 22:28:53.606271 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:53.606227 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="636d5b64-a287-4a2b-bee0-db3b0184b0f1" path="/var/lib/kubelet/pods/636d5b64-a287-4a2b-bee0-db3b0184b0f1/volumes" Oct 30 22:28:54.524663 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:54.524345 6006 generic.go:356] "Generic (PLEG): container finished" podID="f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4" containerID="bc943006670f01789c0d71f38305737d128cf76e94c18b958a990e99c24cf7bc" exitCode=0 Oct 30 22:28:54.524663 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:54.524407 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5" event={"ID":"f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4","Type":"ContainerDied","Data":"bc943006670f01789c0d71f38305737d128cf76e94c18b958a990e99c24cf7bc"} Oct 30 22:28:55.505519 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.505489 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:28:55.538866 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.538827 6006 generic.go:356] "Generic (PLEG): container finished" podID="f9c79304-5ef5-4fc9-ba42-facc6c16f2b0" containerID="f8c67a25f196c2cfb0fc6182c9ad8f31bf9995776e1d2f5226c0a7662cc7973c" exitCode=0 Oct 30 22:28:55.539418 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.538881 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" Oct 30 22:28:55.539418 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.538910 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" event={"ID":"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0","Type":"ContainerDied","Data":"f8c67a25f196c2cfb0fc6182c9ad8f31bf9995776e1d2f5226c0a7662cc7973c"} Oct 30 22:28:55.539418 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.538971 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp" event={"ID":"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0","Type":"ContainerDied","Data":"b54252bbbdd8bd3bb16d4fd81f37c847d9fb7af69e7ad94470909f2cb1d8225c"} Oct 30 22:28:55.539418 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.539003 6006 scope.go:122] "RemoveContainer" containerID="f8c67a25f196c2cfb0fc6182c9ad8f31bf9995776e1d2f5226c0a7662cc7973c" Oct 30 22:28:55.553491 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.553469 6006 scope.go:122] "RemoveContainer" containerID="b588614d2050bf7f198d97ec8d7765e9acd0373a689feab40d3cd18ba44d2d47" Oct 30 22:28:55.566125 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.566100 6006 scope.go:122] "RemoveContainer" containerID="f8c67a25f196c2cfb0fc6182c9ad8f31bf9995776e1d2f5226c0a7662cc7973c" Oct 30 22:28:55.566570 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:28:55.566542 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c67a25f196c2cfb0fc6182c9ad8f31bf9995776e1d2f5226c0a7662cc7973c\": container with ID starting with f8c67a25f196c2cfb0fc6182c9ad8f31bf9995776e1d2f5226c0a7662cc7973c not found: ID does not exist" containerID="f8c67a25f196c2cfb0fc6182c9ad8f31bf9995776e1d2f5226c0a7662cc7973c" Oct 30 22:28:55.566693 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.566576 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c67a25f196c2cfb0fc6182c9ad8f31bf9995776e1d2f5226c0a7662cc7973c"} err="failed to get container status \"f8c67a25f196c2cfb0fc6182c9ad8f31bf9995776e1d2f5226c0a7662cc7973c\": rpc error: code = NotFound desc = could not find container \"f8c67a25f196c2cfb0fc6182c9ad8f31bf9995776e1d2f5226c0a7662cc7973c\": container with ID starting with f8c67a25f196c2cfb0fc6182c9ad8f31bf9995776e1d2f5226c0a7662cc7973c not found: ID does not exist" Oct 30 22:28:55.566693 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.566596 6006 scope.go:122] "RemoveContainer" containerID="b588614d2050bf7f198d97ec8d7765e9acd0373a689feab40d3cd18ba44d2d47" Oct 30 22:28:55.566949 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:28:55.566931 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b588614d2050bf7f198d97ec8d7765e9acd0373a689feab40d3cd18ba44d2d47\": container with ID starting with b588614d2050bf7f198d97ec8d7765e9acd0373a689feab40d3cd18ba44d2d47 not found: ID does not exist" containerID="b588614d2050bf7f198d97ec8d7765e9acd0373a689feab40d3cd18ba44d2d47" Oct 30 22:28:55.566995 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.566962 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b588614d2050bf7f198d97ec8d7765e9acd0373a689feab40d3cd18ba44d2d47"} err="failed to get container status \"b588614d2050bf7f198d97ec8d7765e9acd0373a689feab40d3cd18ba44d2d47\": rpc error: code = NotFound desc = could not find container \"b588614d2050bf7f198d97ec8d7765e9acd0373a689feab40d3cd18ba44d2d47\": container with ID starting with b588614d2050bf7f198d97ec8d7765e9acd0373a689feab40d3cd18ba44d2d47 not found: ID does not exist" Oct 30 22:28:55.646336 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.642882 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-envoy\") pod \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " Oct 30 22:28:55.646336 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.642969 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-kube-api-access-msk6z\" (UniqueName: \"kubernetes.io/projected/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-kube-api-access-msk6z\") pod \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " Oct 30 22:28:55.646336 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.643023 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-workload-socket\") pod \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " Oct 30 22:28:55.646336 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.643055 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istiod-ca-cert\") pod \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " Oct 30 22:28:55.646336 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.643178 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-token\" (UniqueName: \"kubernetes.io/projected/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-token\") pod \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " Oct 30 22:28:55.646336 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.643268 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-credential-socket\") pod \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " Oct 30 22:28:55.646336 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.643330 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-podinfo\") pod \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " Oct 30 22:28:55.646336 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.643361 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-workload-certs\") pod \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " Oct 30 22:28:55.646336 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.643399 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-data\") pod \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\" (UID: \"f9c79304-5ef5-4fc9-ba42-facc6c16f2b0\") " Oct 30 22:28:55.646336 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.644337 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-data" pod "f9c79304-5ef5-4fc9-ba42-facc6c16f2b0" (UID: "f9c79304-5ef5-4fc9-ba42-facc6c16f2b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:55.646336 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.645260 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-workload-socket" pod "f9c79304-5ef5-4fc9-ba42-facc6c16f2b0" (UID: "f9c79304-5ef5-4fc9-ba42-facc6c16f2b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:55.657461 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.652042 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-kube-api-access-msk6z" pod "f9c79304-5ef5-4fc9-ba42-facc6c16f2b0" (UID: "f9c79304-5ef5-4fc9-ba42-facc6c16f2b0"). InnerVolumeSpecName "kube-api-access-msk6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:55.657461 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.652447 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istiod-ca-cert" pod "f9c79304-5ef5-4fc9-ba42-facc6c16f2b0" (UID: "f9c79304-5ef5-4fc9-ba42-facc6c16f2b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:55.657461 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.652697 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-credential-socket" pod "f9c79304-5ef5-4fc9-ba42-facc6c16f2b0" (UID: "f9c79304-5ef5-4fc9-ba42-facc6c16f2b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:55.657461 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.652888 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-workload-certs" pod "f9c79304-5ef5-4fc9-ba42-facc6c16f2b0" (UID: "f9c79304-5ef5-4fc9-ba42-facc6c16f2b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:55.657461 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.653515 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-envoy" pod "f9c79304-5ef5-4fc9-ba42-facc6c16f2b0" (UID: "f9c79304-5ef5-4fc9-ba42-facc6c16f2b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:55.657461 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.656241 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-podinfo" pod "f9c79304-5ef5-4fc9-ba42-facc6c16f2b0" (UID: "f9c79304-5ef5-4fc9-ba42-facc6c16f2b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:28:55.658039 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.657891 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-token" pod "f9c79304-5ef5-4fc9-ba42-facc6c16f2b0" (UID: "f9c79304-5ef5-4fc9-ba42-facc6c16f2b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:55.745345 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.745210 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:55.745345 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.745265 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:55.745345 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.745283 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:55.745345 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.745297 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:55.745345 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.745310 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:55.745345 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.745324 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msk6z\" (UniqueName: \"kubernetes.io/projected/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-kube-api-access-msk6z\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:55.745345 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.745338 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:55.745345 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.745353 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:55.745849 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.745367 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:55.753808 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.753780 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5" Oct 30 22:28:55.852597 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.851811 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4-maas-api-openshift-ca\") pod \"f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4\" (UID: \"f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4\") " Oct 30 22:28:55.852597 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.851995 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4-maas-api-xks-ca\") pod \"f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4\" (UID: \"f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4\") " Oct 30 22:28:55.852597 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.852078 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4-kube-api-access-dfnph\" (UniqueName: \"kubernetes.io/projected/f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4-kube-api-access-dfnph\") pod \"f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4\" (UID: \"f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4\") " Oct 30 22:28:55.862546 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.858957 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4-maas-api-openshift-ca" pod "f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4" (UID: "f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:55.862546 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.861655 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4-maas-api-xks-ca" pod "f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4" (UID: "f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:55.862546 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.862431 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4-kube-api-access-dfnph" pod "f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4" (UID: "f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4"). InnerVolumeSpecName "kube-api-access-dfnph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:55.953812 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.953770 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:55.953812 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.953805 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:55.953812 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.953816 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfnph\" (UniqueName: \"kubernetes.io/projected/f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4-kube-api-access-dfnph\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:55.995774 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:55.995690 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp"] Oct 30 22:28:56.003410 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:56.003376 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d7cdcc-openshift-default-7795c85cf9-rg5bp"] Oct 30 22:28:56.042124 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:56.042077 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw"] Oct 30 22:28:56.045062 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:56.045032 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cf1826-qhsrw"] Oct 30 22:28:56.551663 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:56.551630 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5" Oct 30 22:28:56.552142 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:56.551667 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5" event={"ID":"f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4","Type":"ContainerDied","Data":"75392af1af5992bae2a0181c046ca942fa1d8acee0da5d88c8491adaae112b40"} Oct 30 22:28:56.552142 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:56.551716 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75392af1af5992bae2a0181c046ca942fa1d8acee0da5d88c8491adaae112b40" Oct 30 22:28:57.394741 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.394714 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:28:57.487769 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.487731 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-workload-certs\") pod \"0ec694be-68a9-4227-96ac-cf7daac3af60\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " Oct 30 22:28:57.487953 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.487804 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ec694be-68a9-4227-96ac-cf7daac3af60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ec694be-68a9-4227-96ac-cf7daac3af60-istiod-ca-cert\") pod \"0ec694be-68a9-4227-96ac-cf7daac3af60\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " Oct 30 22:28:57.487953 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.487870 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-podinfo\") pod \"0ec694be-68a9-4227-96ac-cf7daac3af60\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " Oct 30 22:28:57.487953 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.487940 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-token\" (UniqueName: \"kubernetes.io/projected/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-token\") pod \"0ec694be-68a9-4227-96ac-cf7daac3af60\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " Oct 30 22:28:57.488066 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.488016 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-workload-certs" pod "0ec694be-68a9-4227-96ac-cf7daac3af60" (UID: "0ec694be-68a9-4227-96ac-cf7daac3af60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:57.488066 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.488054 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-workload-socket\") pod \"0ec694be-68a9-4227-96ac-cf7daac3af60\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " Oct 30 22:28:57.488169 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.488080 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-data\") pod \"0ec694be-68a9-4227-96ac-cf7daac3af60\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " Oct 30 22:28:57.488169 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.488111 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-credential-socket\") pod \"0ec694be-68a9-4227-96ac-cf7daac3af60\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " Oct 30 22:28:57.488302 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.488198 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-envoy\") pod \"0ec694be-68a9-4227-96ac-cf7daac3af60\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " Oct 30 22:28:57.488353 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.488325 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec694be-68a9-4227-96ac-cf7daac3af60-istiod-ca-cert" pod "0ec694be-68a9-4227-96ac-cf7daac3af60" (UID: "0ec694be-68a9-4227-96ac-cf7daac3af60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:57.488509 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.488476 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-credential-socket" pod "0ec694be-68a9-4227-96ac-cf7daac3af60" (UID: "0ec694be-68a9-4227-96ac-cf7daac3af60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:57.488509 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.488502 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-data" pod "0ec694be-68a9-4227-96ac-cf7daac3af60" (UID: "0ec694be-68a9-4227-96ac-cf7daac3af60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:57.488704 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.488557 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-workload-socket" pod "0ec694be-68a9-4227-96ac-cf7daac3af60" (UID: "0ec694be-68a9-4227-96ac-cf7daac3af60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:57.488704 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.488586 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec694be-68a9-4227-96ac-cf7daac3af60-kube-api-access-mt7dl\" (UniqueName: \"kubernetes.io/projected/0ec694be-68a9-4227-96ac-cf7daac3af60-kube-api-access-mt7dl\") pod \"0ec694be-68a9-4227-96ac-cf7daac3af60\" (UID: \"0ec694be-68a9-4227-96ac-cf7daac3af60\") " Oct 30 22:28:57.489275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.489219 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:57.489275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.489242 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ec694be-68a9-4227-96ac-cf7daac3af60-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:57.489275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.489272 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:57.489515 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.489285 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:57.489515 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.489298 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:57.490473 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.490451 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-token" pod "0ec694be-68a9-4227-96ac-cf7daac3af60" (UID: "0ec694be-68a9-4227-96ac-cf7daac3af60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:57.490732 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.490710 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-podinfo" pod "0ec694be-68a9-4227-96ac-cf7daac3af60" (UID: "0ec694be-68a9-4227-96ac-cf7daac3af60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:28:57.490793 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.490705 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-envoy" pod "0ec694be-68a9-4227-96ac-cf7daac3af60" (UID: "0ec694be-68a9-4227-96ac-cf7daac3af60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:57.490924 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.490905 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec694be-68a9-4227-96ac-cf7daac3af60-kube-api-access-mt7dl" pod "0ec694be-68a9-4227-96ac-cf7daac3af60" (UID: "0ec694be-68a9-4227-96ac-cf7daac3af60"). InnerVolumeSpecName "kube-api-access-mt7dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:57.563740 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.563699 6006 generic.go:356] "Generic (PLEG): container finished" podID="0ec694be-68a9-4227-96ac-cf7daac3af60" containerID="09efbed3d73532097e2c8cc48b361c1b72d96e02302e6a9927708c8f929606e5" exitCode=0 Oct 30 22:28:57.564223 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.563755 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" Oct 30 22:28:57.564223 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.563780 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" event={"ID":"0ec694be-68a9-4227-96ac-cf7daac3af60","Type":"ContainerDied","Data":"09efbed3d73532097e2c8cc48b361c1b72d96e02302e6a9927708c8f929606e5"} Oct 30 22:28:57.564223 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.563845 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx" event={"ID":"0ec694be-68a9-4227-96ac-cf7daac3af60","Type":"ContainerDied","Data":"316ff7f83b9bbff55fc4e418c16a7feaf2eca15d4da8cc09ad9e0e15cdf967ba"} Oct 30 22:28:57.564223 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.563871 6006 scope.go:122] "RemoveContainer" containerID="09efbed3d73532097e2c8cc48b361c1b72d96e02302e6a9927708c8f929606e5" Oct 30 22:28:57.577327 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.577291 6006 scope.go:122] "RemoveContainer" containerID="839dfd2571d546fa1f815a462ff1a8cadf8afee3f65d81846b4a4f7885cf1dcb" Oct 30 22:28:57.587906 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.587881 6006 scope.go:122] "RemoveContainer" containerID="09efbed3d73532097e2c8cc48b361c1b72d96e02302e6a9927708c8f929606e5" Oct 30 22:28:57.588340 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:28:57.588301 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09efbed3d73532097e2c8cc48b361c1b72d96e02302e6a9927708c8f929606e5\": container with ID starting with 09efbed3d73532097e2c8cc48b361c1b72d96e02302e6a9927708c8f929606e5 not found: ID does not exist" containerID="09efbed3d73532097e2c8cc48b361c1b72d96e02302e6a9927708c8f929606e5" Oct 30 22:28:57.588430 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.588357 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09efbed3d73532097e2c8cc48b361c1b72d96e02302e6a9927708c8f929606e5"} err="failed to get container status \"09efbed3d73532097e2c8cc48b361c1b72d96e02302e6a9927708c8f929606e5\": rpc error: code = NotFound desc = could not find container \"09efbed3d73532097e2c8cc48b361c1b72d96e02302e6a9927708c8f929606e5\": container with ID starting with 09efbed3d73532097e2c8cc48b361c1b72d96e02302e6a9927708c8f929606e5 not found: ID does not exist" Oct 30 22:28:57.588430 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.588383 6006 scope.go:122] "RemoveContainer" containerID="839dfd2571d546fa1f815a462ff1a8cadf8afee3f65d81846b4a4f7885cf1dcb" Oct 30 22:28:57.588743 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:28:57.588721 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"839dfd2571d546fa1f815a462ff1a8cadf8afee3f65d81846b4a4f7885cf1dcb\": container with ID starting with 839dfd2571d546fa1f815a462ff1a8cadf8afee3f65d81846b4a4f7885cf1dcb not found: ID does not exist" containerID="839dfd2571d546fa1f815a462ff1a8cadf8afee3f65d81846b4a4f7885cf1dcb" Oct 30 22:28:57.588835 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.588748 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"839dfd2571d546fa1f815a462ff1a8cadf8afee3f65d81846b4a4f7885cf1dcb"} err="failed to get container status \"839dfd2571d546fa1f815a462ff1a8cadf8afee3f65d81846b4a4f7885cf1dcb\": rpc error: code = NotFound desc = could not find container \"839dfd2571d546fa1f815a462ff1a8cadf8afee3f65d81846b4a4f7885cf1dcb\": container with ID starting with 839dfd2571d546fa1f815a462ff1a8cadf8afee3f65d81846b4a4f7885cf1dcb not found: ID does not exist" Oct 30 22:28:57.590586 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.590563 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:57.590662 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.590591 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:57.590662 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.590606 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ec694be-68a9-4227-96ac-cf7daac3af60-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:57.590662 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.590620 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt7dl\" (UniqueName: \"kubernetes.io/projected/0ec694be-68a9-4227-96ac-cf7daac3af60-kube-api-access-mt7dl\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:57.606996 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.606966 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51b0ff5c-2571-4224-9ed5-d40513e49ae5" path="/var/lib/kubelet/pods/51b0ff5c-2571-4224-9ed5-d40513e49ae5/volumes" Oct 30 22:28:57.607664 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.607645 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9c79304-5ef5-4fc9-ba42-facc6c16f2b0" path="/var/lib/kubelet/pods/f9c79304-5ef5-4fc9-ba42-facc6c16f2b0/volumes" Oct 30 22:28:57.722913 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.722833 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx"] Oct 30 22:28:57.728504 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:57.728470 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0b6a3f-openshift-default-57457cb97-h72lx"] Oct 30 22:28:57.919863 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:28:57.919817 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:28:59.246306 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.246199 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58"] Oct 30 22:28:59.247014 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.246990 6006 cpu_manager.go:414] "containerMap: removing container" podUID="f9c79304-5ef5-4fc9-ba42-facc6c16f2b0" containerName="istio-proxy" Oct 30 22:28:59.247113 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.247016 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c79304-5ef5-4fc9-ba42-facc6c16f2b0" containerName="istio-proxy" Oct 30 22:28:59.247113 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.247038 6006 cpu_manager.go:414] "containerMap: removing container" podUID="0ec694be-68a9-4227-96ac-cf7daac3af60" containerName="istio-proxy" Oct 30 22:28:59.247113 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.247045 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec694be-68a9-4227-96ac-cf7daac3af60" containerName="istio-proxy" Oct 30 22:28:59.247113 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.247073 6006 cpu_manager.go:414] "containerMap: removing container" podUID="f9c79304-5ef5-4fc9-ba42-facc6c16f2b0" containerName="istio-proxy" Oct 30 22:28:59.247113 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.247083 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c79304-5ef5-4fc9-ba42-facc6c16f2b0" containerName="istio-proxy" Oct 30 22:28:59.247113 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.247096 6006 cpu_manager.go:414] "containerMap: removing container" podUID="f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4" containerName="revoke-keys" Oct 30 22:28:59.247113 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.247103 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4" containerName="revoke-keys" Oct 30 22:28:59.247470 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.247287 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c79304-5ef5-4fc9-ba42-facc6c16f2b0" containerName="istio-proxy" Oct 30 22:28:59.247470 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.247303 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ec694be-68a9-4227-96ac-cf7daac3af60" containerName="istio-proxy" Oct 30 22:28:59.247470 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.247316 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c79304-5ef5-4fc9-ba42-facc6c16f2b0" containerName="istio-proxy" Oct 30 22:28:59.247470 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.247327 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ec694be-68a9-4227-96ac-cf7daac3af60" containerName="istio-proxy" Oct 30 22:28:59.247470 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.247336 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4" containerName="revoke-keys" Oct 30 22:28:59.252949 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.252920 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58" Oct 30 22:28:59.303157 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.303118 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:28:59.308058 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.307982 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58"] Oct 30 22:28:59.349074 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.349034 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:28:59.418132 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.418095 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98826100-a8f5-4845-b5c8-e3684cee6ec5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58\" (UID: \"98826100-a8f5-4845-b5c8-e3684cee6ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58" Oct 30 22:28:59.418351 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.418171 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6bjq\" (UniqueName: \"kubernetes.io/projected/98826100-a8f5-4845-b5c8-e3684cee6ec5-kube-api-access-f6bjq\") pod \"maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58\" (UID: \"98826100-a8f5-4845-b5c8-e3684cee6ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58" Oct 30 22:28:59.418351 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.418288 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98826100-a8f5-4845-b5c8-e3684cee6ec5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58\" (UID: \"98826100-a8f5-4845-b5c8-e3684cee6ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58" Oct 30 22:28:59.519942 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.519835 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98826100-a8f5-4845-b5c8-e3684cee6ec5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58\" (UID: \"98826100-a8f5-4845-b5c8-e3684cee6ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58" Oct 30 22:28:59.520136 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.519956 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6bjq\" (UniqueName: \"kubernetes.io/projected/98826100-a8f5-4845-b5c8-e3684cee6ec5-kube-api-access-f6bjq\") pod \"maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58\" (UID: \"98826100-a8f5-4845-b5c8-e3684cee6ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58" Oct 30 22:28:59.520136 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.519994 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98826100-a8f5-4845-b5c8-e3684cee6ec5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58\" (UID: \"98826100-a8f5-4845-b5c8-e3684cee6ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58" Oct 30 22:28:59.520752 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.520724 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98826100-a8f5-4845-b5c8-e3684cee6ec5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58\" (UID: \"98826100-a8f5-4845-b5c8-e3684cee6ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58" Oct 30 22:28:59.522505 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.522475 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98826100-a8f5-4845-b5c8-e3684cee6ec5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58\" (UID: \"98826100-a8f5-4845-b5c8-e3684cee6ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58" Oct 30 22:28:59.579268 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.579229 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6bjq\" (UniqueName: \"kubernetes.io/projected/98826100-a8f5-4845-b5c8-e3684cee6ec5-kube-api-access-f6bjq\") pod \"maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58\" (UID: \"98826100-a8f5-4845-b5c8-e3684cee6ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58" Oct 30 22:28:59.606057 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.606020 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ec694be-68a9-4227-96ac-cf7daac3af60" path="/var/lib/kubelet/pods/0ec694be-68a9-4227-96ac-cf7daac3af60/volumes" Oct 30 22:28:59.720449 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.720406 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2" podUID="9a510562-a7e8-4abe-963e-3ddd4ff88162" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:28:59.866570 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:28:59.866532 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58" Oct 30 22:29:00.267340 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:00.267308 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58"] Oct 30 22:29:00.269554 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:29:00.269527 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98826100_a8f5_4845_b5c8_e3684cee6ec5.slice/crio-b22a480b785aec20c3745f362c541067dcd8b8e42d3ab6e5e15ffe6f7085d666 WatchSource:0}: Error finding container b22a480b785aec20c3745f362c541067dcd8b8e42d3ab6e5e15ffe6f7085d666: Status 404 returned error can't find the container with id b22a480b785aec20c3745f362c541067dcd8b8e42d3ab6e5e15ffe6f7085d666 Oct 30 22:29:00.596500 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:00.596453 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58" event={"ID":"98826100-a8f5-4845-b5c8-e3684cee6ec5","Type":"ContainerStarted","Data":"b22a480b785aec20c3745f362c541067dcd8b8e42d3ab6e5e15ffe6f7085d666"} Oct 30 22:29:01.620138 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:01.620029 6006 generic.go:356] "Generic (PLEG): container finished" podID="98826100-a8f5-4845-b5c8-e3684cee6ec5" containerID="95043e8d8433543ad8098d67d9e79837bbdddd960c68ba90a8ae94ecb309224f" exitCode=0 Oct 30 22:29:01.620550 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:01.620127 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58" event={"ID":"98826100-a8f5-4845-b5c8-e3684cee6ec5","Type":"ContainerDied","Data":"95043e8d8433543ad8098d67d9e79837bbdddd960c68ba90a8ae94ecb309224f"} Oct 30 22:29:02.795754 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:02.795723 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58" Oct 30 22:29:02.976003 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:02.967767 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98826100-a8f5-4845-b5c8-e3684cee6ec5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98826100-a8f5-4845-b5c8-e3684cee6ec5-maas-api-xks-ca\") pod \"98826100-a8f5-4845-b5c8-e3684cee6ec5\" (UID: \"98826100-a8f5-4845-b5c8-e3684cee6ec5\") " Oct 30 22:29:02.976583 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:02.976401 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98826100-a8f5-4845-b5c8-e3684cee6ec5-kube-api-access-f6bjq\" (UniqueName: \"kubernetes.io/projected/98826100-a8f5-4845-b5c8-e3684cee6ec5-kube-api-access-f6bjq\") pod \"98826100-a8f5-4845-b5c8-e3684cee6ec5\" (UID: \"98826100-a8f5-4845-b5c8-e3684cee6ec5\") " Oct 30 22:29:02.979682 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:02.979646 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98826100-a8f5-4845-b5c8-e3684cee6ec5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98826100-a8f5-4845-b5c8-e3684cee6ec5-maas-api-openshift-ca\") pod \"98826100-a8f5-4845-b5c8-e3684cee6ec5\" (UID: \"98826100-a8f5-4845-b5c8-e3684cee6ec5\") " Oct 30 22:29:02.980183 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:02.980157 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98826100-a8f5-4845-b5c8-e3684cee6ec5-maas-api-openshift-ca" pod "98826100-a8f5-4845-b5c8-e3684cee6ec5" (UID: "98826100-a8f5-4845-b5c8-e3684cee6ec5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:02.980552 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:02.980524 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98826100-a8f5-4845-b5c8-e3684cee6ec5-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:03.038661 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:03.033410 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98826100-a8f5-4845-b5c8-e3684cee6ec5-maas-api-xks-ca" pod "98826100-a8f5-4845-b5c8-e3684cee6ec5" (UID: "98826100-a8f5-4845-b5c8-e3684cee6ec5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:29:03.038661 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:03.034840 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98826100-a8f5-4845-b5c8-e3684cee6ec5-kube-api-access-f6bjq" pod "98826100-a8f5-4845-b5c8-e3684cee6ec5" (UID: "98826100-a8f5-4845-b5c8-e3684cee6ec5"). InnerVolumeSpecName "kube-api-access-f6bjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:03.082154 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:03.082112 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6bjq\" (UniqueName: \"kubernetes.io/projected/98826100-a8f5-4845-b5c8-e3684cee6ec5-kube-api-access-f6bjq\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:03.082154 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:03.082152 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98826100-a8f5-4845-b5c8-e3684cee6ec5-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:03.636899 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:03.636866 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58" Oct 30 22:29:03.637103 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:03.636898 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58" event={"ID":"98826100-a8f5-4845-b5c8-e3684cee6ec5","Type":"ContainerDied","Data":"b22a480b785aec20c3745f362c541067dcd8b8e42d3ab6e5e15ffe6f7085d666"} Oct 30 22:29:03.637103 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:03.636960 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b22a480b785aec20c3745f362c541067dcd8b8e42d3ab6e5e15ffe6f7085d666" Oct 30 22:29:05.035230 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:05.035190 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6"] Oct 30 22:29:05.035822 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:05.035805 6006 cpu_manager.go:414] "containerMap: removing container" podUID="0ec694be-68a9-4227-96ac-cf7daac3af60" containerName="istio-proxy" Oct 30 22:29:05.035872 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:05.035823 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec694be-68a9-4227-96ac-cf7daac3af60" containerName="istio-proxy" Oct 30 22:29:05.035872 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:05.035835 6006 cpu_manager.go:414] "containerMap: removing container" podUID="98826100-a8f5-4845-b5c8-e3684cee6ec5" containerName="revoke-keys" Oct 30 22:29:05.035872 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:05.035841 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98826100-a8f5-4845-b5c8-e3684cee6ec5" containerName="revoke-keys" Oct 30 22:29:05.035968 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:05.035948 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="98826100-a8f5-4845-b5c8-e3684cee6ec5" containerName="revoke-keys" Oct 30 22:29:05.050428 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:05.042541 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6" Oct 30 22:29:05.096918 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:05.096880 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6"] Oct 30 22:29:05.128591 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:05.128549 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8dacf80-a617-41ac-ba6f-8c84fe6b6e90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6\" (UID: \"f8dacf80-a617-41ac-ba6f-8c84fe6b6e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6" Oct 30 22:29:05.128769 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:05.128628 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j94q\" (UniqueName: \"kubernetes.io/projected/f8dacf80-a617-41ac-ba6f-8c84fe6b6e90-kube-api-access-2j94q\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6\" (UID: \"f8dacf80-a617-41ac-ba6f-8c84fe6b6e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6" Oct 30 22:29:05.128858 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:05.128839 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8dacf80-a617-41ac-ba6f-8c84fe6b6e90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6\" (UID: \"f8dacf80-a617-41ac-ba6f-8c84fe6b6e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6" Oct 30 22:29:05.138606 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:05.138548 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:29:05.138794 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:05.138634 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:29:05.230006 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:05.229965 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8dacf80-a617-41ac-ba6f-8c84fe6b6e90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6\" (UID: \"f8dacf80-a617-41ac-ba6f-8c84fe6b6e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6" Oct 30 22:29:05.230330 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:05.230082 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8dacf80-a617-41ac-ba6f-8c84fe6b6e90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6\" (UID: \"f8dacf80-a617-41ac-ba6f-8c84fe6b6e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6" Oct 30 22:29:05.230330 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:05.230122 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j94q\" (UniqueName: \"kubernetes.io/projected/f8dacf80-a617-41ac-ba6f-8c84fe6b6e90-kube-api-access-2j94q\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6\" (UID: \"f8dacf80-a617-41ac-ba6f-8c84fe6b6e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6" Oct 30 22:29:05.230754 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:05.230723 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8dacf80-a617-41ac-ba6f-8c84fe6b6e90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6\" (UID: \"f8dacf80-a617-41ac-ba6f-8c84fe6b6e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6" Oct 30 22:29:05.232724 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:05.232702 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8dacf80-a617-41ac-ba6f-8c84fe6b6e90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6\" (UID: \"f8dacf80-a617-41ac-ba6f-8c84fe6b6e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6" Oct 30 22:29:05.289374 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:05.289304 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j94q\" (UniqueName: \"kubernetes.io/projected/f8dacf80-a617-41ac-ba6f-8c84fe6b6e90-kube-api-access-2j94q\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6\" (UID: \"f8dacf80-a617-41ac-ba6f-8c84fe6b6e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6" Oct 30 22:29:05.390158 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:05.390122 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6" Oct 30 22:29:05.867171 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:05.867140 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6"] Oct 30 22:29:05.868785 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:29:05.868740 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8dacf80_a617_41ac_ba6f_8c84fe6b6e90.slice/crio-ef06b0ff9f04eaf3adf5c112f4956962b47c2928f0a07b2f1338619b4c83759c WatchSource:0}: Error finding container ef06b0ff9f04eaf3adf5c112f4956962b47c2928f0a07b2f1338619b4c83759c: Status 404 returned error can't find the container with id ef06b0ff9f04eaf3adf5c112f4956962b47c2928f0a07b2f1338619b4c83759c Oct 30 22:29:06.662613 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:06.662570 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6" event={"ID":"f8dacf80-a617-41ac-ba6f-8c84fe6b6e90","Type":"ContainerStarted","Data":"776d84e7a34c181bb80ea5fc9c1d7ae41bfe7e4a338ed36a1ccc9e8efd36509e"} Oct 30 22:29:06.663067 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:06.662634 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6" event={"ID":"f8dacf80-a617-41ac-ba6f-8c84fe6b6e90","Type":"ContainerStarted","Data":"ef06b0ff9f04eaf3adf5c112f4956962b47c2928f0a07b2f1338619b4c83759c"} Oct 30 22:29:06.774530 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:06.774473 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6" podStartSLOduration=2.77445889 podStartE2EDuration="2.77445889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:06.771959875 +0000 UTC m=+2349.324136199" watchObservedRunningTime="2026-10-30 22:29:06.77445889 +0000 UTC m=+2349.326635237" Oct 30 22:29:07.672043 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:07.672006 6006 generic.go:356] "Generic (PLEG): container finished" podID="f8dacf80-a617-41ac-ba6f-8c84fe6b6e90" containerID="776d84e7a34c181bb80ea5fc9c1d7ae41bfe7e4a338ed36a1ccc9e8efd36509e" exitCode=0 Oct 30 22:29:07.672508 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:07.672095 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6" event={"ID":"f8dacf80-a617-41ac-ba6f-8c84fe6b6e90","Type":"ContainerDied","Data":"776d84e7a34c181bb80ea5fc9c1d7ae41bfe7e4a338ed36a1ccc9e8efd36509e"} Oct 30 22:29:08.864645 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:08.864619 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6" Oct 30 22:29:08.982032 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:08.981948 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8dacf80-a617-41ac-ba6f-8c84fe6b6e90-kube-api-access-2j94q\" (UniqueName: \"kubernetes.io/projected/f8dacf80-a617-41ac-ba6f-8c84fe6b6e90-kube-api-access-2j94q\") pod \"f8dacf80-a617-41ac-ba6f-8c84fe6b6e90\" (UID: \"f8dacf80-a617-41ac-ba6f-8c84fe6b6e90\") " Oct 30 22:29:08.982188 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:08.982006 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8dacf80-a617-41ac-ba6f-8c84fe6b6e90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8dacf80-a617-41ac-ba6f-8c84fe6b6e90-maas-api-xks-ca\") pod \"f8dacf80-a617-41ac-ba6f-8c84fe6b6e90\" (UID: \"f8dacf80-a617-41ac-ba6f-8c84fe6b6e90\") " Oct 30 22:29:08.982188 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:08.982147 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8dacf80-a617-41ac-ba6f-8c84fe6b6e90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8dacf80-a617-41ac-ba6f-8c84fe6b6e90-maas-api-openshift-ca\") pod \"f8dacf80-a617-41ac-ba6f-8c84fe6b6e90\" (UID: \"f8dacf80-a617-41ac-ba6f-8c84fe6b6e90\") " Oct 30 22:29:08.982603 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:08.982573 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8dacf80-a617-41ac-ba6f-8c84fe6b6e90-maas-api-openshift-ca" pod "f8dacf80-a617-41ac-ba6f-8c84fe6b6e90" (UID: "f8dacf80-a617-41ac-ba6f-8c84fe6b6e90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:08.982730 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:08.982672 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8dacf80-a617-41ac-ba6f-8c84fe6b6e90-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:08.984380 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:08.984352 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8dacf80-a617-41ac-ba6f-8c84fe6b6e90-maas-api-xks-ca" pod "f8dacf80-a617-41ac-ba6f-8c84fe6b6e90" (UID: "f8dacf80-a617-41ac-ba6f-8c84fe6b6e90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:29:08.984477 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:08.984376 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8dacf80-a617-41ac-ba6f-8c84fe6b6e90-kube-api-access-2j94q" pod "f8dacf80-a617-41ac-ba6f-8c84fe6b6e90" (UID: "f8dacf80-a617-41ac-ba6f-8c84fe6b6e90"). InnerVolumeSpecName "kube-api-access-2j94q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:09.083689 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:09.083648 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j94q\" (UniqueName: \"kubernetes.io/projected/f8dacf80-a617-41ac-ba6f-8c84fe6b6e90-kube-api-access-2j94q\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:09.083689 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:09.083684 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8dacf80-a617-41ac-ba6f-8c84fe6b6e90-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:09.360601 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:09.360575 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2" Oct 30 22:29:09.388722 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:09.388687 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a510562-a7e8-4abe-963e-3ddd4ff88162-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a510562-a7e8-4abe-963e-3ddd4ff88162-plugins-config-volume\") pod \"9a510562-a7e8-4abe-963e-3ddd4ff88162\" (UID: \"9a510562-a7e8-4abe-963e-3ddd4ff88162\") " Oct 30 22:29:09.388907 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:09.388761 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a510562-a7e8-4abe-963e-3ddd4ff88162-kube-api-access-27wrv\" (UniqueName: \"kubernetes.io/projected/9a510562-a7e8-4abe-963e-3ddd4ff88162-kube-api-access-27wrv\") pod \"9a510562-a7e8-4abe-963e-3ddd4ff88162\" (UID: \"9a510562-a7e8-4abe-963e-3ddd4ff88162\") " Oct 30 22:29:09.389157 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:09.389123 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a510562-a7e8-4abe-963e-3ddd4ff88162-plugins-config-volume" pod "9a510562-a7e8-4abe-963e-3ddd4ff88162" (UID: "9a510562-a7e8-4abe-963e-3ddd4ff88162"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:09.389500 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:09.389481 6006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a510562-a7e8-4abe-963e-3ddd4ff88162-plugins-config-volume\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:09.390962 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:09.390934 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a510562-a7e8-4abe-963e-3ddd4ff88162-kube-api-access-27wrv" pod "9a510562-a7e8-4abe-963e-3ddd4ff88162" (UID: "9a510562-a7e8-4abe-963e-3ddd4ff88162"). InnerVolumeSpecName "kube-api-access-27wrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:09.490928 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:09.490879 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27wrv\" (UniqueName: \"kubernetes.io/projected/9a510562-a7e8-4abe-963e-3ddd4ff88162-kube-api-access-27wrv\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:09.694679 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:09.694637 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd"] Oct 30 22:29:09.695223 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:09.695179 6006 generic.go:356] "Generic (PLEG): container finished" podID="9a510562-a7e8-4abe-963e-3ddd4ff88162" containerID="f435bc68dd004f99ef69ad5739aca3634b4343eff946ebef1231e5ecffa6c2d9" exitCode=0 Oct 30 22:29:09.695595 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:09.695572 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2" event={"ID":"9a510562-a7e8-4abe-963e-3ddd4ff88162","Type":"ContainerDied","Data":"f435bc68dd004f99ef69ad5739aca3634b4343eff946ebef1231e5ecffa6c2d9"} Oct 30 22:29:09.695791 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:09.695757 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2" event={"ID":"9a510562-a7e8-4abe-963e-3ddd4ff88162","Type":"ContainerDied","Data":"6e196f9ea9c4dd2a8b1f79c8a7d09419ccd9e2298fa02473e4e6c1f591eeef48"} Oct 30 22:29:09.696325 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:09.695948 6006 scope.go:122] "RemoveContainer" containerID="f435bc68dd004f99ef69ad5739aca3634b4343eff946ebef1231e5ecffa6c2d9" Oct 30 22:29:09.696599 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:09.696039 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2" Oct 30 22:29:09.698776 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:09.698456 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6" event={"ID":"f8dacf80-a617-41ac-ba6f-8c84fe6b6e90","Type":"ContainerDied","Data":"ef06b0ff9f04eaf3adf5c112f4956962b47c2928f0a07b2f1338619b4c83759c"} Oct 30 22:29:09.698776 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:09.698494 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6" Oct 30 22:29:09.698776 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:09.698511 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef06b0ff9f04eaf3adf5c112f4956962b47c2928f0a07b2f1338619b4c83759c" Oct 30 22:29:09.710385 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:09.709551 6006 scope.go:122] "RemoveContainer" containerID="f435bc68dd004f99ef69ad5739aca3634b4343eff946ebef1231e5ecffa6c2d9" Oct 30 22:29:09.710385 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:29:09.709887 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f435bc68dd004f99ef69ad5739aca3634b4343eff946ebef1231e5ecffa6c2d9\": container with ID starting with f435bc68dd004f99ef69ad5739aca3634b4343eff946ebef1231e5ecffa6c2d9 not found: ID does not exist" containerID="f435bc68dd004f99ef69ad5739aca3634b4343eff946ebef1231e5ecffa6c2d9" Oct 30 22:29:09.710385 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:09.709917 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f435bc68dd004f99ef69ad5739aca3634b4343eff946ebef1231e5ecffa6c2d9"} err="failed to get container status \"f435bc68dd004f99ef69ad5739aca3634b4343eff946ebef1231e5ecffa6c2d9\": rpc error: code = NotFound desc = could not find container \"f435bc68dd004f99ef69ad5739aca3634b4343eff946ebef1231e5ecffa6c2d9\": container with ID starting with f435bc68dd004f99ef69ad5739aca3634b4343eff946ebef1231e5ecffa6c2d9 not found: ID does not exist" Oct 30 22:29:09.914851 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:09.914755 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2"] Oct 30 22:29:09.920713 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:09.920674 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-65e59f-6c747f7f99-9jjr2"] Oct 30 22:29:11.605045 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:11.605009 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a510562-a7e8-4abe-963e-3ddd4ff88162" path="/var/lib/kubelet/pods/9a510562-a7e8-4abe-963e-3ddd4ff88162/volumes" Oct 30 22:29:14.588846 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:14.588806 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58"] Oct 30 22:29:14.633486 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:14.633450 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-595fd7d7-qpp58"] Oct 30 22:29:14.696027 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:14.695980 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd" podUID="eece1c74-3904-47a7-ab7f-3464ffd6767e" containerName="payload-processing" containerID="cri-o://5ca86708cf567144143f532781ac56dad0377d42873692a1322867a47342bcbd" gracePeriod=25 Oct 30 22:29:15.036273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:15.030607 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd" Oct 30 22:29:15.207088 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:15.206988 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eece1c74-3904-47a7-ab7f-3464ffd6767e-kube-api-access-frzth\" (UniqueName: \"kubernetes.io/projected/eece1c74-3904-47a7-ab7f-3464ffd6767e-kube-api-access-frzth\") pod \"eece1c74-3904-47a7-ab7f-3464ffd6767e\" (UID: \"eece1c74-3904-47a7-ab7f-3464ffd6767e\") " Oct 30 22:29:15.207309 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:15.207143 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eece1c74-3904-47a7-ab7f-3464ffd6767e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eece1c74-3904-47a7-ab7f-3464ffd6767e-plugins-config-volume\") pod \"eece1c74-3904-47a7-ab7f-3464ffd6767e\" (UID: \"eece1c74-3904-47a7-ab7f-3464ffd6767e\") " Oct 30 22:29:15.207570 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:15.207545 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eece1c74-3904-47a7-ab7f-3464ffd6767e-plugins-config-volume" pod "eece1c74-3904-47a7-ab7f-3464ffd6767e" (UID: "eece1c74-3904-47a7-ab7f-3464ffd6767e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:15.209304 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:15.209277 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eece1c74-3904-47a7-ab7f-3464ffd6767e-kube-api-access-frzth" pod "eece1c74-3904-47a7-ab7f-3464ffd6767e" (UID: "eece1c74-3904-47a7-ab7f-3464ffd6767e"). InnerVolumeSpecName "kube-api-access-frzth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:15.309330 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:15.309296 6006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eece1c74-3904-47a7-ab7f-3464ffd6767e-plugins-config-volume\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:15.309330 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:15.309330 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frzth\" (UniqueName: \"kubernetes.io/projected/eece1c74-3904-47a7-ab7f-3464ffd6767e-kube-api-access-frzth\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:15.606052 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:15.606009 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98826100-a8f5-4845-b5c8-e3684cee6ec5" path="/var/lib/kubelet/pods/98826100-a8f5-4845-b5c8-e3684cee6ec5/volumes" Oct 30 22:29:15.750804 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:15.750764 6006 generic.go:356] "Generic (PLEG): container finished" podID="eece1c74-3904-47a7-ab7f-3464ffd6767e" containerID="5ca86708cf567144143f532781ac56dad0377d42873692a1322867a47342bcbd" exitCode=0 Oct 30 22:29:15.750963 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:15.750812 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd" Oct 30 22:29:15.750963 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:15.750848 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd" event={"ID":"eece1c74-3904-47a7-ab7f-3464ffd6767e","Type":"ContainerDied","Data":"5ca86708cf567144143f532781ac56dad0377d42873692a1322867a47342bcbd"} Oct 30 22:29:15.750963 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:15.750914 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd" event={"ID":"eece1c74-3904-47a7-ab7f-3464ffd6767e","Type":"ContainerDied","Data":"d3aa89ed44da3497c944057fa153abf9bb536eee5aa58b3758512569bb4fad19"} Oct 30 22:29:15.750963 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:15.750943 6006 scope.go:122] "RemoveContainer" containerID="5ca86708cf567144143f532781ac56dad0377d42873692a1322867a47342bcbd" Oct 30 22:29:15.762704 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:15.762676 6006 scope.go:122] "RemoveContainer" containerID="5ca86708cf567144143f532781ac56dad0377d42873692a1322867a47342bcbd" Oct 30 22:29:15.763031 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:29:15.763009 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ca86708cf567144143f532781ac56dad0377d42873692a1322867a47342bcbd\": container with ID starting with 5ca86708cf567144143f532781ac56dad0377d42873692a1322867a47342bcbd not found: ID does not exist" containerID="5ca86708cf567144143f532781ac56dad0377d42873692a1322867a47342bcbd" Oct 30 22:29:15.763111 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:15.763044 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ca86708cf567144143f532781ac56dad0377d42873692a1322867a47342bcbd"} err="failed to get container status \"5ca86708cf567144143f532781ac56dad0377d42873692a1322867a47342bcbd\": rpc error: code = NotFound desc = could not find container \"5ca86708cf567144143f532781ac56dad0377d42873692a1322867a47342bcbd\": container with ID starting with 5ca86708cf567144143f532781ac56dad0377d42873692a1322867a47342bcbd not found: ID does not exist" Oct 30 22:29:15.911403 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:15.911323 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd"] Oct 30 22:29:15.960087 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:15.960045 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-595fd7d7-8b7867549-jbpqd"] Oct 30 22:29:17.605462 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:17.605430 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eece1c74-3904-47a7-ab7f-3464ffd6767e" path="/var/lib/kubelet/pods/eece1c74-3904-47a7-ab7f-3464ffd6767e/volumes" Oct 30 22:29:22.092783 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:22.092747 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6"] Oct 30 22:29:22.093390 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:22.092962 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" podUID="e020ac39-b5d1-4657-9b2b-ce1a06e4c988" containerName="istio-proxy" containerID="cri-o://608b0c0a3f78a7055564dc3582544787e187a2f6626774880c141336c61340b6" gracePeriod=30 Oct 30 22:29:27.385788 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.385754 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:29:27.510412 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.510377 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-podinfo\") pod \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " Oct 30 22:29:27.510646 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.510442 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-data\") pod \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " Oct 30 22:29:27.510646 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.510466 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-kube-api-access-dblk2\" (UniqueName: \"kubernetes.io/projected/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-kube-api-access-dblk2\") pod \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " Oct 30 22:29:27.510646 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.510553 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-workload-socket\") pod \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " Oct 30 22:29:27.510646 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.510573 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-workload-certs\") pod \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " Oct 30 22:29:27.510646 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.510607 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-envoy\") pod \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " Oct 30 22:29:27.510909 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.510760 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-token\" (UniqueName: \"kubernetes.io/projected/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-token\") pod \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " Oct 30 22:29:27.510909 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.510796 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-credential-socket\") pod \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " Oct 30 22:29:27.510909 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.510902 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istiod-ca-cert\") pod \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\" (UID: \"e020ac39-b5d1-4657-9b2b-ce1a06e4c988\") " Oct 30 22:29:27.511065 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.510755 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-data" pod "e020ac39-b5d1-4657-9b2b-ce1a06e4c988" (UID: "e020ac39-b5d1-4657-9b2b-ce1a06e4c988"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:27.511065 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.510963 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-workload-certs" pod "e020ac39-b5d1-4657-9b2b-ce1a06e4c988" (UID: "e020ac39-b5d1-4657-9b2b-ce1a06e4c988"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:27.511174 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.511088 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-workload-socket" pod "e020ac39-b5d1-4657-9b2b-ce1a06e4c988" (UID: "e020ac39-b5d1-4657-9b2b-ce1a06e4c988"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:27.511415 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.511394 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-credential-socket" pod "e020ac39-b5d1-4657-9b2b-ce1a06e4c988" (UID: "e020ac39-b5d1-4657-9b2b-ce1a06e4c988"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:27.511516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.511424 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istiod-ca-cert" pod "e020ac39-b5d1-4657-9b2b-ce1a06e4c988" (UID: "e020ac39-b5d1-4657-9b2b-ce1a06e4c988"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:27.511643 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.511624 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:27.511699 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.511649 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:27.511699 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.511664 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:27.511699 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.511676 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:27.511699 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.511689 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:27.513511 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.513479 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-token" pod "e020ac39-b5d1-4657-9b2b-ce1a06e4c988" (UID: "e020ac39-b5d1-4657-9b2b-ce1a06e4c988"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:27.513631 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.513588 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-envoy" pod "e020ac39-b5d1-4657-9b2b-ce1a06e4c988" (UID: "e020ac39-b5d1-4657-9b2b-ce1a06e4c988"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:27.513677 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.513652 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-podinfo" pod "e020ac39-b5d1-4657-9b2b-ce1a06e4c988" (UID: "e020ac39-b5d1-4657-9b2b-ce1a06e4c988"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:29:27.513837 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.513813 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-kube-api-access-dblk2" pod "e020ac39-b5d1-4657-9b2b-ce1a06e4c988" (UID: "e020ac39-b5d1-4657-9b2b-ce1a06e4c988"). InnerVolumeSpecName "kube-api-access-dblk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:27.612597 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.612558 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:27.612597 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.612593 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:27.612597 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.612606 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:27.612884 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.612621 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dblk2\" (UniqueName: \"kubernetes.io/projected/e020ac39-b5d1-4657-9b2b-ce1a06e4c988-kube-api-access-dblk2\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:27.885191 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.885110 6006 generic.go:356] "Generic (PLEG): container finished" podID="e020ac39-b5d1-4657-9b2b-ce1a06e4c988" containerID="608b0c0a3f78a7055564dc3582544787e187a2f6626774880c141336c61340b6" exitCode=0 Oct 30 22:29:27.885191 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.885166 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" Oct 30 22:29:27.885191 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.885174 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" event={"ID":"e020ac39-b5d1-4657-9b2b-ce1a06e4c988","Type":"ContainerDied","Data":"608b0c0a3f78a7055564dc3582544787e187a2f6626774880c141336c61340b6"} Oct 30 22:29:27.885468 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.885217 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6" event={"ID":"e020ac39-b5d1-4657-9b2b-ce1a06e4c988","Type":"ContainerDied","Data":"95b349dc0fd962e54459d5b888cc4ed8dfb6fd3fc7d465d2119cd6edc44dcea2"} Oct 30 22:29:27.885468 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.885242 6006 scope.go:122] "RemoveContainer" containerID="608b0c0a3f78a7055564dc3582544787e187a2f6626774880c141336c61340b6" Oct 30 22:29:27.913002 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.912975 6006 scope.go:122] "RemoveContainer" containerID="989bcae7013c3d8510aa0f5888ff3ed850c95eaad116bc27c5d9bb33ce63ed01" Oct 30 22:29:27.920497 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:29:27.920455 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:29:27.926478 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.926450 6006 scope.go:122] "RemoveContainer" containerID="608b0c0a3f78a7055564dc3582544787e187a2f6626774880c141336c61340b6" Oct 30 22:29:27.926854 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:29:27.926827 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"608b0c0a3f78a7055564dc3582544787e187a2f6626774880c141336c61340b6\": container with ID starting with 608b0c0a3f78a7055564dc3582544787e187a2f6626774880c141336c61340b6 not found: ID does not exist" containerID="608b0c0a3f78a7055564dc3582544787e187a2f6626774880c141336c61340b6" Oct 30 22:29:27.926967 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.926868 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"608b0c0a3f78a7055564dc3582544787e187a2f6626774880c141336c61340b6"} err="failed to get container status \"608b0c0a3f78a7055564dc3582544787e187a2f6626774880c141336c61340b6\": rpc error: code = NotFound desc = could not find container \"608b0c0a3f78a7055564dc3582544787e187a2f6626774880c141336c61340b6\": container with ID starting with 608b0c0a3f78a7055564dc3582544787e187a2f6626774880c141336c61340b6 not found: ID does not exist" Oct 30 22:29:27.926967 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.926897 6006 scope.go:122] "RemoveContainer" containerID="989bcae7013c3d8510aa0f5888ff3ed850c95eaad116bc27c5d9bb33ce63ed01" Oct 30 22:29:27.927243 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:29:27.927220 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"989bcae7013c3d8510aa0f5888ff3ed850c95eaad116bc27c5d9bb33ce63ed01\": container with ID starting with 989bcae7013c3d8510aa0f5888ff3ed850c95eaad116bc27c5d9bb33ce63ed01 not found: ID does not exist" containerID="989bcae7013c3d8510aa0f5888ff3ed850c95eaad116bc27c5d9bb33ce63ed01" Oct 30 22:29:27.927367 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:27.927294 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"989bcae7013c3d8510aa0f5888ff3ed850c95eaad116bc27c5d9bb33ce63ed01"} err="failed to get container status \"989bcae7013c3d8510aa0f5888ff3ed850c95eaad116bc27c5d9bb33ce63ed01\": rpc error: code = NotFound desc = could not find container \"989bcae7013c3d8510aa0f5888ff3ed850c95eaad116bc27c5d9bb33ce63ed01\": container with ID starting with 989bcae7013c3d8510aa0f5888ff3ed850c95eaad116bc27c5d9bb33ce63ed01 not found: ID does not exist" Oct 30 22:29:28.046920 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:28.046811 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6"] Oct 30 22:29:28.093067 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:28.093028 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-595fd7d7-openshift-default-55f55bfd65-srfq6"] Oct 30 22:29:29.609031 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:29.608989 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e020ac39-b5d1-4657-9b2b-ce1a06e4c988" path="/var/lib/kubelet/pods/e020ac39-b5d1-4657-9b2b-ce1a06e4c988/volumes" Oct 30 22:29:30.330024 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:30.329988 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:29:31.533104 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.533067 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92"] Oct 30 22:29:31.533668 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.533651 6006 cpu_manager.go:414] "containerMap: removing container" podUID="e020ac39-b5d1-4657-9b2b-ce1a06e4c988" containerName="istio-proxy" Oct 30 22:29:31.533726 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.533671 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e020ac39-b5d1-4657-9b2b-ce1a06e4c988" containerName="istio-proxy" Oct 30 22:29:31.533726 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.533690 6006 cpu_manager.go:414] "containerMap: removing container" podUID="eece1c74-3904-47a7-ab7f-3464ffd6767e" containerName="payload-processing" Oct 30 22:29:31.533726 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.533696 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eece1c74-3904-47a7-ab7f-3464ffd6767e" containerName="payload-processing" Oct 30 22:29:31.533726 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.533715 6006 cpu_manager.go:414] "containerMap: removing container" podUID="f8dacf80-a617-41ac-ba6f-8c84fe6b6e90" containerName="revoke-keys" Oct 30 22:29:31.533726 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.533720 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8dacf80-a617-41ac-ba6f-8c84fe6b6e90" containerName="revoke-keys" Oct 30 22:29:31.533902 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.533730 6006 cpu_manager.go:414] "containerMap: removing container" podUID="e020ac39-b5d1-4657-9b2b-ce1a06e4c988" containerName="istio-proxy" Oct 30 22:29:31.533902 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.533740 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e020ac39-b5d1-4657-9b2b-ce1a06e4c988" containerName="istio-proxy" Oct 30 22:29:31.533902 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.533785 6006 cpu_manager.go:414] "containerMap: removing container" podUID="9a510562-a7e8-4abe-963e-3ddd4ff88162" containerName="payload-processing" Oct 30 22:29:31.533902 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.533794 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a510562-a7e8-4abe-963e-3ddd4ff88162" containerName="payload-processing" Oct 30 22:29:31.534036 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.533932 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="e020ac39-b5d1-4657-9b2b-ce1a06e4c988" containerName="istio-proxy" Oct 30 22:29:31.534036 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.533946 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="eece1c74-3904-47a7-ab7f-3464ffd6767e" containerName="payload-processing" Oct 30 22:29:31.534036 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.533957 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8dacf80-a617-41ac-ba6f-8c84fe6b6e90" containerName="revoke-keys" Oct 30 22:29:31.534036 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.533963 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="e020ac39-b5d1-4657-9b2b-ce1a06e4c988" containerName="istio-proxy" Oct 30 22:29:31.534036 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.533974 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a510562-a7e8-4abe-963e-3ddd4ff88162" containerName="payload-processing" Oct 30 22:29:31.539171 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.539146 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92" Oct 30 22:29:31.594124 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.594079 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92"] Oct 30 22:29:31.635831 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.635794 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-582593\"" Oct 30 22:29:31.636036 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.636017 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-582593-dockercfg-t9bjg\"" Oct 30 22:29:31.665947 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.665896 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkvxx\" (UniqueName: \"kubernetes.io/projected/c8a428e4-1d5b-452d-a330-5d9c22c7d1f5-kube-api-access-zkvxx\") pod \"payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92\" (UID: \"c8a428e4-1d5b-452d-a330-5d9c22c7d1f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92" Oct 30 22:29:31.666233 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.666207 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8a428e4-1d5b-452d-a330-5d9c22c7d1f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92\" (UID: \"c8a428e4-1d5b-452d-a330-5d9c22c7d1f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92" Oct 30 22:29:31.767372 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.767331 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8a428e4-1d5b-452d-a330-5d9c22c7d1f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92\" (UID: \"c8a428e4-1d5b-452d-a330-5d9c22c7d1f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92" Oct 30 22:29:31.767568 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.767413 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkvxx\" (UniqueName: \"kubernetes.io/projected/c8a428e4-1d5b-452d-a330-5d9c22c7d1f5-kube-api-access-zkvxx\") pod \"payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92\" (UID: \"c8a428e4-1d5b-452d-a330-5d9c22c7d1f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92" Oct 30 22:29:31.768923 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.768892 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8a428e4-1d5b-452d-a330-5d9c22c7d1f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92\" (UID: \"c8a428e4-1d5b-452d-a330-5d9c22c7d1f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92" Oct 30 22:29:31.828820 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.828784 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkvxx\" (UniqueName: \"kubernetes.io/projected/c8a428e4-1d5b-452d-a330-5d9c22c7d1f5-kube-api-access-zkvxx\") pod \"payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92\" (UID: \"c8a428e4-1d5b-452d-a330-5d9c22c7d1f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92" Oct 30 22:29:31.854828 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:31.854744 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92" Oct 30 22:29:32.348271 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:32.346378 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92"] Oct 30 22:29:32.366865 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:29:32.366830 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a428e4_1d5b_452d_a330_5d9c22c7d1f5.slice/crio-bc52ac1c84ecd404a2fcc70f01e592089df597f499151fb214ff3d39c5714950 WatchSource:0}: Error finding container bc52ac1c84ecd404a2fcc70f01e592089df597f499151fb214ff3d39c5714950: Status 404 returned error can't find the container with id bc52ac1c84ecd404a2fcc70f01e592089df597f499151fb214ff3d39c5714950 Oct 30 22:29:32.979241 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:32.979203 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92" event={"ID":"c8a428e4-1d5b-452d-a330-5d9c22c7d1f5","Type":"ContainerStarted","Data":"0fe175fb65c0dd00ee202ef571ed3bba3b246bfa858cf2edee03e6ed05959b73"} Oct 30 22:29:32.979687 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:32.979289 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92" Oct 30 22:29:32.979687 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:32.979316 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92" event={"ID":"c8a428e4-1d5b-452d-a330-5d9c22c7d1f5","Type":"ContainerStarted","Data":"bc52ac1c84ecd404a2fcc70f01e592089df597f499151fb214ff3d39c5714950"} Oct 30 22:29:33.104901 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:33.104844 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92" podStartSLOduration=2.104830667 podStartE2EDuration="2.104830667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:33.102682647 +0000 UTC m=+2375.654858948" watchObservedRunningTime="2026-10-30 22:29:33.104830667 +0000 UTC m=+2375.657006967" Oct 30 22:29:33.466557 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:33.466520 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5"] Oct 30 22:29:33.469361 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:33.469334 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-3cc1b3-bd9b5"] Oct 30 22:29:33.604220 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:33.604175 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4" path="/var/lib/kubelet/pods/f49457d4-4c5f-41f2-b8d1-bbdb4357a6d4/volumes" Oct 30 22:29:41.437755 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:41.437711 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq"] Oct 30 22:29:41.443506 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:41.443478 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq" Oct 30 22:29:41.496573 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:41.496538 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq"] Oct 30 22:29:41.538892 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:41.538855 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:29:41.538892 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:41.538859 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:29:41.621348 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:41.621308 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9033ff6b-3ba3-42cd-b5e5-3dc557a05acb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq\" (UID: \"9033ff6b-3ba3-42cd-b5e5-3dc557a05acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq" Oct 30 22:29:41.621562 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:41.621486 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lhkz\" (UniqueName: \"kubernetes.io/projected/9033ff6b-3ba3-42cd-b5e5-3dc557a05acb-kube-api-access-4lhkz\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq\" (UID: \"9033ff6b-3ba3-42cd-b5e5-3dc557a05acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq" Oct 30 22:29:41.621562 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:41.621549 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9033ff6b-3ba3-42cd-b5e5-3dc557a05acb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq\" (UID: \"9033ff6b-3ba3-42cd-b5e5-3dc557a05acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq" Oct 30 22:29:41.722913 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:41.722808 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9033ff6b-3ba3-42cd-b5e5-3dc557a05acb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq\" (UID: \"9033ff6b-3ba3-42cd-b5e5-3dc557a05acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq" Oct 30 22:29:41.722913 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:41.722887 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lhkz\" (UniqueName: \"kubernetes.io/projected/9033ff6b-3ba3-42cd-b5e5-3dc557a05acb-kube-api-access-4lhkz\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq\" (UID: \"9033ff6b-3ba3-42cd-b5e5-3dc557a05acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq" Oct 30 22:29:41.722913 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:41.722917 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9033ff6b-3ba3-42cd-b5e5-3dc557a05acb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq\" (UID: \"9033ff6b-3ba3-42cd-b5e5-3dc557a05acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq" Oct 30 22:29:41.723539 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:41.723515 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9033ff6b-3ba3-42cd-b5e5-3dc557a05acb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq\" (UID: \"9033ff6b-3ba3-42cd-b5e5-3dc557a05acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq" Oct 30 22:29:41.725535 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:41.725505 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9033ff6b-3ba3-42cd-b5e5-3dc557a05acb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq\" (UID: \"9033ff6b-3ba3-42cd-b5e5-3dc557a05acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq" Oct 30 22:29:41.781870 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:41.781839 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lhkz\" (UniqueName: \"kubernetes.io/projected/9033ff6b-3ba3-42cd-b5e5-3dc557a05acb-kube-api-access-4lhkz\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq\" (UID: \"9033ff6b-3ba3-42cd-b5e5-3dc557a05acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq" Oct 30 22:29:42.058627 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:42.058597 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq" Oct 30 22:29:42.437896 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:42.437857 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq"] Oct 30 22:29:42.440126 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:29:42.440078 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9033ff6b_3ba3_42cd_b5e5_3dc557a05acb.slice/crio-20c3b1e7ce8ea1c812642e6f9f6e60f2503ef1809fc6df08b37b3904d073bc27 WatchSource:0}: Error finding container 20c3b1e7ce8ea1c812642e6f9f6e60f2503ef1809fc6df08b37b3904d073bc27: Status 404 returned error can't find the container with id 20c3b1e7ce8ea1c812642e6f9f6e60f2503ef1809fc6df08b37b3904d073bc27 Oct 30 22:29:43.055575 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:43.055525 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq" event={"ID":"9033ff6b-3ba3-42cd-b5e5-3dc557a05acb","Type":"ContainerStarted","Data":"b7f7e44e09945fd8e5ad2160a0804761ebd263c00b1c4790fff25c77cc19d06b"} Oct 30 22:29:43.055783 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:43.055594 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq" event={"ID":"9033ff6b-3ba3-42cd-b5e5-3dc557a05acb","Type":"ContainerStarted","Data":"20c3b1e7ce8ea1c812642e6f9f6e60f2503ef1809fc6df08b37b3904d073bc27"} Oct 30 22:29:43.167998 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:43.167896 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq" podStartSLOduration=2.167881222 podStartE2EDuration="2.167881222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:43.164780893 +0000 UTC m=+2385.716957179" watchObservedRunningTime="2026-10-30 22:29:43.167881222 +0000 UTC m=+2385.720057524" Oct 30 22:29:43.990087 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:43.990053 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92" Oct 30 22:29:44.066643 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:44.066607 6006 generic.go:356] "Generic (PLEG): container finished" podID="9033ff6b-3ba3-42cd-b5e5-3dc557a05acb" containerID="b7f7e44e09945fd8e5ad2160a0804761ebd263c00b1c4790fff25c77cc19d06b" exitCode=0 Oct 30 22:29:44.066850 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:44.066697 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq" event={"ID":"9033ff6b-3ba3-42cd-b5e5-3dc557a05acb","Type":"ContainerDied","Data":"b7f7e44e09945fd8e5ad2160a0804761ebd263c00b1c4790fff25c77cc19d06b"} Oct 30 22:29:45.240241 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:45.240208 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq" Oct 30 22:29:45.269070 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:45.269030 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9033ff6b-3ba3-42cd-b5e5-3dc557a05acb-kube-api-access-4lhkz\" (UniqueName: \"kubernetes.io/projected/9033ff6b-3ba3-42cd-b5e5-3dc557a05acb-kube-api-access-4lhkz\") pod \"9033ff6b-3ba3-42cd-b5e5-3dc557a05acb\" (UID: \"9033ff6b-3ba3-42cd-b5e5-3dc557a05acb\") " Oct 30 22:29:45.269260 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:45.269114 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9033ff6b-3ba3-42cd-b5e5-3dc557a05acb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9033ff6b-3ba3-42cd-b5e5-3dc557a05acb-maas-api-openshift-ca\") pod \"9033ff6b-3ba3-42cd-b5e5-3dc557a05acb\" (UID: \"9033ff6b-3ba3-42cd-b5e5-3dc557a05acb\") " Oct 30 22:29:45.269260 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:45.269190 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9033ff6b-3ba3-42cd-b5e5-3dc557a05acb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9033ff6b-3ba3-42cd-b5e5-3dc557a05acb-maas-api-xks-ca\") pod \"9033ff6b-3ba3-42cd-b5e5-3dc557a05acb\" (UID: \"9033ff6b-3ba3-42cd-b5e5-3dc557a05acb\") " Oct 30 22:29:45.269542 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:45.269516 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9033ff6b-3ba3-42cd-b5e5-3dc557a05acb-maas-api-openshift-ca" pod "9033ff6b-3ba3-42cd-b5e5-3dc557a05acb" (UID: "9033ff6b-3ba3-42cd-b5e5-3dc557a05acb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:45.271311 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:45.271283 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9033ff6b-3ba3-42cd-b5e5-3dc557a05acb-kube-api-access-4lhkz" pod "9033ff6b-3ba3-42cd-b5e5-3dc557a05acb" (UID: "9033ff6b-3ba3-42cd-b5e5-3dc557a05acb"). InnerVolumeSpecName "kube-api-access-4lhkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:45.271440 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:45.271417 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9033ff6b-3ba3-42cd-b5e5-3dc557a05acb-maas-api-xks-ca" pod "9033ff6b-3ba3-42cd-b5e5-3dc557a05acb" (UID: "9033ff6b-3ba3-42cd-b5e5-3dc557a05acb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:29:45.371268 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:45.371173 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9033ff6b-3ba3-42cd-b5e5-3dc557a05acb-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:45.371268 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:45.371207 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9033ff6b-3ba3-42cd-b5e5-3dc557a05acb-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:45.371268 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:45.371217 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lhkz\" (UniqueName: \"kubernetes.io/projected/9033ff6b-3ba3-42cd-b5e5-3dc557a05acb-kube-api-access-4lhkz\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:46.082901 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:46.082867 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq" Oct 30 22:29:46.083120 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:46.082898 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq" event={"ID":"9033ff6b-3ba3-42cd-b5e5-3dc557a05acb","Type":"ContainerDied","Data":"20c3b1e7ce8ea1c812642e6f9f6e60f2503ef1809fc6df08b37b3904d073bc27"} Oct 30 22:29:46.083120 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:46.082951 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20c3b1e7ce8ea1c812642e6f9f6e60f2503ef1809fc6df08b37b3904d073bc27" Oct 30 22:29:49.706083 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:49.706046 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk"] Oct 30 22:29:49.752481 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:49.752443 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7bb1c2-tp7gk"] Oct 30 22:29:51.604204 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:51.604168 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc" path="/var/lib/kubelet/pods/8ed7dda0-ac0c-4216-bc73-d0cdb53f68dc/volumes" Oct 30 22:29:53.812688 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:53.812647 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e6f692b-wb4ls"] Oct 30 22:29:53.813223 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:53.813209 6006 cpu_manager.go:414] "containerMap: removing container" podUID="9033ff6b-3ba3-42cd-b5e5-3dc557a05acb" containerName="revoke-keys" Oct 30 22:29:53.813295 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:53.813224 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9033ff6b-3ba3-42cd-b5e5-3dc557a05acb" containerName="revoke-keys" Oct 30 22:29:53.813393 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:53.813381 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="9033ff6b-3ba3-42cd-b5e5-3dc557a05acb" containerName="revoke-keys" Oct 30 22:29:53.817049 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:53.817021 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6f692b-wb4ls" Oct 30 22:29:53.871347 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:53.871307 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1c51b57-kf674"] Oct 30 22:29:53.876707 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:53.876675 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c51b57-kf674" Oct 30 22:29:53.895412 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:53.895375 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dac4ba6-533e-4bba-89ed-5679379d5f62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e6f692b-wb4ls\" (UID: \"8dac4ba6-533e-4bba-89ed-5679379d5f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6f692b-wb4ls" Oct 30 22:29:53.895677 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:53.895462 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cgc2\" (UniqueName: \"kubernetes.io/projected/8dac4ba6-533e-4bba-89ed-5679379d5f62-kube-api-access-8cgc2\") pod \"maas-api-revoke-sub-7e6f692b-wb4ls\" (UID: \"8dac4ba6-533e-4bba-89ed-5679379d5f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6f692b-wb4ls" Oct 30 22:29:53.895677 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:53.895550 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dac4ba6-533e-4bba-89ed-5679379d5f62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e6f692b-wb4ls\" (UID: \"8dac4ba6-533e-4bba-89ed-5679379d5f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6f692b-wb4ls" Oct 30 22:29:53.915318 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:53.915212 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:29:53.915559 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:53.915239 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:29:53.918294 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:53.918262 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e6f692b-wb4ls"] Oct 30 22:29:53.937684 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:53.937652 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1c51b57-kf674"] Oct 30 22:29:53.997183 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:53.997146 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cgc2\" (UniqueName: \"kubernetes.io/projected/8dac4ba6-533e-4bba-89ed-5679379d5f62-kube-api-access-8cgc2\") pod \"maas-api-revoke-sub-7e6f692b-wb4ls\" (UID: \"8dac4ba6-533e-4bba-89ed-5679379d5f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6f692b-wb4ls" Oct 30 22:29:53.997446 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:53.997266 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dac4ba6-533e-4bba-89ed-5679379d5f62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e6f692b-wb4ls\" (UID: \"8dac4ba6-533e-4bba-89ed-5679379d5f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6f692b-wb4ls" Oct 30 22:29:53.997446 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:53.997322 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hsx4\" (UniqueName: \"kubernetes.io/projected/400871ab-5b3b-4b9c-8893-75e692a1e551-kube-api-access-8hsx4\") pod \"maas-api-revoke-sub-f1c51b57-kf674\" (UID: \"400871ab-5b3b-4b9c-8893-75e692a1e551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c51b57-kf674" Oct 30 22:29:53.997557 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:53.997437 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400871ab-5b3b-4b9c-8893-75e692a1e551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1c51b57-kf674\" (UID: \"400871ab-5b3b-4b9c-8893-75e692a1e551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c51b57-kf674" Oct 30 22:29:53.997557 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:53.997527 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400871ab-5b3b-4b9c-8893-75e692a1e551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1c51b57-kf674\" (UID: \"400871ab-5b3b-4b9c-8893-75e692a1e551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c51b57-kf674" Oct 30 22:29:53.997717 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:53.997696 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dac4ba6-533e-4bba-89ed-5679379d5f62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e6f692b-wb4ls\" (UID: \"8dac4ba6-533e-4bba-89ed-5679379d5f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6f692b-wb4ls" Oct 30 22:29:53.998536 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:53.998504 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dac4ba6-533e-4bba-89ed-5679379d5f62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e6f692b-wb4ls\" (UID: \"8dac4ba6-533e-4bba-89ed-5679379d5f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6f692b-wb4ls" Oct 30 22:29:53.999991 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:53.999966 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dac4ba6-533e-4bba-89ed-5679379d5f62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e6f692b-wb4ls\" (UID: \"8dac4ba6-533e-4bba-89ed-5679379d5f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6f692b-wb4ls" Oct 30 22:29:54.056229 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:54.056194 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cgc2\" (UniqueName: \"kubernetes.io/projected/8dac4ba6-533e-4bba-89ed-5679379d5f62-kube-api-access-8cgc2\") pod \"maas-api-revoke-sub-7e6f692b-wb4ls\" (UID: \"8dac4ba6-533e-4bba-89ed-5679379d5f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6f692b-wb4ls" Oct 30 22:29:54.099713 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:54.099616 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hsx4\" (UniqueName: \"kubernetes.io/projected/400871ab-5b3b-4b9c-8893-75e692a1e551-kube-api-access-8hsx4\") pod \"maas-api-revoke-sub-f1c51b57-kf674\" (UID: \"400871ab-5b3b-4b9c-8893-75e692a1e551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c51b57-kf674" Oct 30 22:29:54.099898 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:54.099725 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400871ab-5b3b-4b9c-8893-75e692a1e551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1c51b57-kf674\" (UID: \"400871ab-5b3b-4b9c-8893-75e692a1e551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c51b57-kf674" Oct 30 22:29:54.099898 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:54.099768 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400871ab-5b3b-4b9c-8893-75e692a1e551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1c51b57-kf674\" (UID: \"400871ab-5b3b-4b9c-8893-75e692a1e551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c51b57-kf674" Oct 30 22:29:54.100716 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:54.100682 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400871ab-5b3b-4b9c-8893-75e692a1e551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1c51b57-kf674\" (UID: \"400871ab-5b3b-4b9c-8893-75e692a1e551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c51b57-kf674" Oct 30 22:29:54.102415 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:54.102382 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400871ab-5b3b-4b9c-8893-75e692a1e551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1c51b57-kf674\" (UID: \"400871ab-5b3b-4b9c-8893-75e692a1e551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c51b57-kf674" Oct 30 22:29:54.132586 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:54.132538 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6f692b-wb4ls" Oct 30 22:29:54.159266 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:54.159224 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hsx4\" (UniqueName: \"kubernetes.io/projected/400871ab-5b3b-4b9c-8893-75e692a1e551-kube-api-access-8hsx4\") pod \"maas-api-revoke-sub-f1c51b57-kf674\" (UID: \"400871ab-5b3b-4b9c-8893-75e692a1e551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c51b57-kf674" Oct 30 22:29:54.197566 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:54.197452 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c51b57-kf674" Oct 30 22:29:54.526862 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:54.526814 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e6f692b-wb4ls"] Oct 30 22:29:54.624389 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:54.624339 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1c51b57-kf674"] Oct 30 22:29:55.155447 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:55.155344 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6f692b-wb4ls" event={"ID":"8dac4ba6-533e-4bba-89ed-5679379d5f62","Type":"ContainerStarted","Data":"72d4e659cac1f60b4f84efc1c361bca3f42b34427a6e48817eae709b21315c72"} Oct 30 22:29:55.155447 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:55.155411 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6f692b-wb4ls" event={"ID":"8dac4ba6-533e-4bba-89ed-5679379d5f62","Type":"ContainerStarted","Data":"5c3b44faac666b780c920dd41867ba85ecfa43ede03b87efa6de0a8a3173cbf8"} Oct 30 22:29:55.157215 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:55.157187 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c51b57-kf674" event={"ID":"400871ab-5b3b-4b9c-8893-75e692a1e551","Type":"ContainerStarted","Data":"18917dd94687da1776b0a4814e7e6205e434862e9332a8d132d1f58bf199b7ee"} Oct 30 22:29:55.157386 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:55.157242 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c51b57-kf674" event={"ID":"400871ab-5b3b-4b9c-8893-75e692a1e551","Type":"ContainerStarted","Data":"1cac743bfb7d9661e66a9d2507c6d5219f96b2fc3930005013053c917273d1c3"} Oct 30 22:29:55.266125 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:55.266070 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6f692b-wb4ls" podStartSLOduration=2.266056764 podStartE2EDuration="2.266056764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:55.264433044 +0000 UTC m=+2397.816609344" watchObservedRunningTime="2026-10-30 22:29:55.266056764 +0000 UTC m=+2397.818233065" Oct 30 22:29:55.374641 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:55.374491 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c51b57-kf674" podStartSLOduration=2.3744726050000002 podStartE2EDuration="2.374472605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:55.373869915 +0000 UTC m=+2397.926046218" watchObservedRunningTime="2026-10-30 22:29:55.374472605 +0000 UTC m=+2397.926648910" Oct 30 22:29:56.169469 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:56.169423 6006 generic.go:356] "Generic (PLEG): container finished" podID="400871ab-5b3b-4b9c-8893-75e692a1e551" containerID="18917dd94687da1776b0a4814e7e6205e434862e9332a8d132d1f58bf199b7ee" exitCode=0 Oct 30 22:29:56.172003 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:56.169606 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c51b57-kf674" event={"ID":"400871ab-5b3b-4b9c-8893-75e692a1e551","Type":"ContainerDied","Data":"18917dd94687da1776b0a4814e7e6205e434862e9332a8d132d1f58bf199b7ee"} Oct 30 22:29:56.172603 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:56.172576 6006 generic.go:356] "Generic (PLEG): container finished" podID="8dac4ba6-533e-4bba-89ed-5679379d5f62" containerID="72d4e659cac1f60b4f84efc1c361bca3f42b34427a6e48817eae709b21315c72" exitCode=0 Oct 30 22:29:56.172729 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:56.172697 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6f692b-wb4ls" event={"ID":"8dac4ba6-533e-4bba-89ed-5679379d5f62","Type":"ContainerDied","Data":"72d4e659cac1f60b4f84efc1c361bca3f42b34427a6e48817eae709b21315c72"} Oct 30 22:29:56.270686 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:56.270649 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9"] Oct 30 22:29:56.270877 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:56.270856 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" podUID="ab56f9ed-7e2f-4a74-8dda-53df36c32f23" containerName="istio-proxy" containerID="cri-o://a4885693acc976444deef6cd022219f61bd92b56374102a38907c95f2a0502a1" gracePeriod=30 Oct 30 22:29:56.384122 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:56.384073 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p"] Oct 30 22:29:56.384365 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:56.384336 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" podUID="f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3" containerName="istio-proxy" containerID="cri-o://009be692572310e3168485877a4133d7ffcab468040cfda7549f62534e356956" gracePeriod=30 Oct 30 22:29:57.396571 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:57.391758 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9"] Oct 30 22:29:57.396571 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:57.392043 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" podUID="f33ebace-e652-4f07-bbce-9c334faa3948" containerName="istio-proxy" containerID="cri-o://a7f28a559a912c74ad8c03267b0ae6427e548a2e7c3a57364b48b76ce5dc6f12" gracePeriod=30 Oct 30 22:29:57.548555 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:57.548456 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c51b57-kf674" Oct 30 22:29:57.582540 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:57.582506 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6f692b-wb4ls" Oct 30 22:29:57.634542 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:57.634509 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/400871ab-5b3b-4b9c-8893-75e692a1e551-kube-api-access-8hsx4\" (UniqueName: \"kubernetes.io/projected/400871ab-5b3b-4b9c-8893-75e692a1e551-kube-api-access-8hsx4\") pod \"400871ab-5b3b-4b9c-8893-75e692a1e551\" (UID: \"400871ab-5b3b-4b9c-8893-75e692a1e551\") " Oct 30 22:29:57.634747 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:57.634567 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dac4ba6-533e-4bba-89ed-5679379d5f62-kube-api-access-8cgc2\" (UniqueName: \"kubernetes.io/projected/8dac4ba6-533e-4bba-89ed-5679379d5f62-kube-api-access-8cgc2\") pod \"8dac4ba6-533e-4bba-89ed-5679379d5f62\" (UID: \"8dac4ba6-533e-4bba-89ed-5679379d5f62\") " Oct 30 22:29:57.634747 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:57.634691 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dac4ba6-533e-4bba-89ed-5679379d5f62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dac4ba6-533e-4bba-89ed-5679379d5f62-maas-api-openshift-ca\") pod \"8dac4ba6-533e-4bba-89ed-5679379d5f62\" (UID: \"8dac4ba6-533e-4bba-89ed-5679379d5f62\") " Oct 30 22:29:57.634871 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:57.634754 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/400871ab-5b3b-4b9c-8893-75e692a1e551-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400871ab-5b3b-4b9c-8893-75e692a1e551-maas-api-xks-ca\") pod \"400871ab-5b3b-4b9c-8893-75e692a1e551\" (UID: \"400871ab-5b3b-4b9c-8893-75e692a1e551\") " Oct 30 22:29:57.634871 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:57.634817 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dac4ba6-533e-4bba-89ed-5679379d5f62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dac4ba6-533e-4bba-89ed-5679379d5f62-maas-api-xks-ca\") pod \"8dac4ba6-533e-4bba-89ed-5679379d5f62\" (UID: \"8dac4ba6-533e-4bba-89ed-5679379d5f62\") " Oct 30 22:29:57.634871 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:57.634852 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/400871ab-5b3b-4b9c-8893-75e692a1e551-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400871ab-5b3b-4b9c-8893-75e692a1e551-maas-api-openshift-ca\") pod \"400871ab-5b3b-4b9c-8893-75e692a1e551\" (UID: \"400871ab-5b3b-4b9c-8893-75e692a1e551\") " Oct 30 22:29:57.635158 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:57.635126 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dac4ba6-533e-4bba-89ed-5679379d5f62-maas-api-openshift-ca" pod "8dac4ba6-533e-4bba-89ed-5679379d5f62" (UID: "8dac4ba6-533e-4bba-89ed-5679379d5f62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:57.635461 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:57.635428 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/400871ab-5b3b-4b9c-8893-75e692a1e551-maas-api-openshift-ca" pod "400871ab-5b3b-4b9c-8893-75e692a1e551" (UID: "400871ab-5b3b-4b9c-8893-75e692a1e551"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:57.635586 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:57.635520 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dac4ba6-533e-4bba-89ed-5679379d5f62-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:57.637095 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:57.637073 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400871ab-5b3b-4b9c-8893-75e692a1e551-kube-api-access-8hsx4" pod "400871ab-5b3b-4b9c-8893-75e692a1e551" (UID: "400871ab-5b3b-4b9c-8893-75e692a1e551"). InnerVolumeSpecName "kube-api-access-8hsx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:57.637261 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:57.637232 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dac4ba6-533e-4bba-89ed-5679379d5f62-kube-api-access-8cgc2" pod "8dac4ba6-533e-4bba-89ed-5679379d5f62" (UID: "8dac4ba6-533e-4bba-89ed-5679379d5f62"). InnerVolumeSpecName "kube-api-access-8cgc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:57.637380 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:57.637354 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/400871ab-5b3b-4b9c-8893-75e692a1e551-maas-api-xks-ca" pod "400871ab-5b3b-4b9c-8893-75e692a1e551" (UID: "400871ab-5b3b-4b9c-8893-75e692a1e551"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:29:57.637470 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:57.637418 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dac4ba6-533e-4bba-89ed-5679379d5f62-maas-api-xks-ca" pod "8dac4ba6-533e-4bba-89ed-5679379d5f62" (UID: "8dac4ba6-533e-4bba-89ed-5679379d5f62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:29:57.736737 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:57.736647 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hsx4\" (UniqueName: \"kubernetes.io/projected/400871ab-5b3b-4b9c-8893-75e692a1e551-kube-api-access-8hsx4\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:57.736737 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:57.736685 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cgc2\" (UniqueName: \"kubernetes.io/projected/8dac4ba6-533e-4bba-89ed-5679379d5f62-kube-api-access-8cgc2\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:57.736737 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:57.736702 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400871ab-5b3b-4b9c-8893-75e692a1e551-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:57.736737 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:57.736718 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dac4ba6-533e-4bba-89ed-5679379d5f62-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:57.736737 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:57.736731 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400871ab-5b3b-4b9c-8893-75e692a1e551-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:57.912239 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:29:57.912197 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:29:58.058637 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:58.058596 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 22:29:58.074164 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:58.074133 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 22:29:58.092400 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:58.092364 6006 patch_prober.go:29] interesting pod/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:29:58.092599 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:58.092423 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" podUID="f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:29:58.188492 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:58.188461 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c51b57-kf674" Oct 30 22:29:58.188663 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:58.188506 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c51b57-kf674" event={"ID":"400871ab-5b3b-4b9c-8893-75e692a1e551","Type":"ContainerDied","Data":"1cac743bfb7d9661e66a9d2507c6d5219f96b2fc3930005013053c917273d1c3"} Oct 30 22:29:58.188663 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:58.188555 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cac743bfb7d9661e66a9d2507c6d5219f96b2fc3930005013053c917273d1c3" Oct 30 22:29:58.189946 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:58.189928 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6f692b-wb4ls" Oct 30 22:29:58.189946 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:58.189932 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6f692b-wb4ls" event={"ID":"8dac4ba6-533e-4bba-89ed-5679379d5f62","Type":"ContainerDied","Data":"5c3b44faac666b780c920dd41867ba85ecfa43ede03b87efa6de0a8a3173cbf8"} Oct 30 22:29:58.190088 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:58.189963 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c3b44faac666b780c920dd41867ba85ecfa43ede03b87efa6de0a8a3173cbf8" Oct 30 22:29:58.236328 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:58.236289 6006 patch_prober.go:29] interesting pod/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:29:58.236504 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:58.236362 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" podUID="ab56f9ed-7e2f-4a74-8dda-53df36c32f23" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:29:59.051565 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:59.051518 6006 scope.go:122] "RemoveContainer" containerID="28b1b1317ac5dfc0ea48b6907219eafc9de4103ece670b7325bdd8f39445e4bb" Oct 30 22:29:59.064713 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:29:59.064629 6006 scope.go:122] "RemoveContainer" containerID="d0e1a0dce780d642d3e38175c21f2a3a8cd2ddc9359859cee77f89b4d82d4bed" Oct 30 22:30:00.177108 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.177065 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs"] Oct 30 22:30:00.177657 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.177636 6006 cpu_manager.go:414] "containerMap: removing container" podUID="8dac4ba6-533e-4bba-89ed-5679379d5f62" containerName="revoke-keys" Oct 30 22:30:00.177657 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.177648 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dac4ba6-533e-4bba-89ed-5679379d5f62" containerName="revoke-keys" Oct 30 22:30:00.177795 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.177664 6006 cpu_manager.go:414] "containerMap: removing container" podUID="400871ab-5b3b-4b9c-8893-75e692a1e551" containerName="revoke-keys" Oct 30 22:30:00.177795 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.177670 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="400871ab-5b3b-4b9c-8893-75e692a1e551" containerName="revoke-keys" Oct 30 22:30:00.177795 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.177774 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dac4ba6-533e-4bba-89ed-5679379d5f62" containerName="revoke-keys" Oct 30 22:30:00.177795 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.177783 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="400871ab-5b3b-4b9c-8893-75e692a1e551" containerName="revoke-keys" Oct 30 22:30:00.183402 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.183296 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs" Oct 30 22:30:00.184582 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.184554 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q"] Oct 30 22:30:00.188636 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.188611 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9"] Oct 30 22:30:00.188773 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.188716 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q" Oct 30 22:30:00.192945 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.192923 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9" Oct 30 22:30:00.223959 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.223927 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg"] Oct 30 22:30:00.228974 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.228944 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5"] Oct 30 22:30:00.229119 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.229034 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg" Oct 30 22:30:00.233945 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.233919 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r"] Oct 30 22:30:00.234467 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.234441 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5" Oct 30 22:30:00.239003 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.238979 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k"] Oct 30 22:30:00.239119 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.239092 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r" Oct 30 22:30:00.243587 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.243564 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k" Oct 30 22:30:00.245699 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.245348 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs"] Oct 30 22:30:00.270531 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.270499 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mln7n\" (UniqueName: \"kubernetes.io/projected/dcc289f1-3dd7-4182-bc07-8ca05b24b13b-kube-api-access-mln7n\") pod \"maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5\" (UID: \"dcc289f1-3dd7-4182-bc07-8ca05b24b13b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5" Oct 30 22:30:00.270771 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.270745 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcc289f1-3dd7-4182-bc07-8ca05b24b13b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5\" (UID: \"dcc289f1-3dd7-4182-bc07-8ca05b24b13b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5" Oct 30 22:30:00.270924 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.270906 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5vbz\" (UniqueName: \"kubernetes.io/projected/34557f8d-a341-4f36-9914-8ec5a9af1228-kube-api-access-m5vbz\") pod \"maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q\" (UID: \"34557f8d-a341-4f36-9914-8ec5a9af1228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q" Oct 30 22:30:00.271194 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.271172 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34557f8d-a341-4f36-9914-8ec5a9af1228-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q\" (UID: \"34557f8d-a341-4f36-9914-8ec5a9af1228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q" Oct 30 22:30:00.271477 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.271456 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlgnn\" (UniqueName: \"kubernetes.io/projected/94535d2d-703e-4804-992a-474908adf7f2-kube-api-access-vlgnn\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs\" (UID: \"94535d2d-703e-4804-992a-474908adf7f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs" Oct 30 22:30:00.271617 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.271601 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km2c2\" (UniqueName: \"kubernetes.io/projected/08446265-b751-4ff4-a788-488684eba3ea-kube-api-access-km2c2\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9\" (UID: \"08446265-b751-4ff4-a788-488684eba3ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9" Oct 30 22:30:00.271729 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.271712 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94535d2d-703e-4804-992a-474908adf7f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs\" (UID: \"94535d2d-703e-4804-992a-474908adf7f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs" Oct 30 22:30:00.271859 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.271844 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08446265-b751-4ff4-a788-488684eba3ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9\" (UID: \"08446265-b751-4ff4-a788-488684eba3ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9" Oct 30 22:30:00.272043 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.272024 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slpbr\" (UniqueName: \"kubernetes.io/projected/cd8cd9b2-105d-4115-a810-44e907efaba4-kube-api-access-slpbr\") pod \"maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg\" (UID: \"cd8cd9b2-105d-4115-a810-44e907efaba4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg" Oct 30 22:30:00.273394 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.273176 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8cd9b2-105d-4115-a810-44e907efaba4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg\" (UID: \"cd8cd9b2-105d-4115-a810-44e907efaba4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg" Oct 30 22:30:00.278587 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.276759 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5"] Oct 30 22:30:00.283938 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.282129 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9"] Oct 30 22:30:00.283938 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.283123 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5" Oct 30 22:30:00.291313 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.287321 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q"] Oct 30 22:30:00.291313 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.287434 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9" Oct 30 22:30:00.296879 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.296855 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:30:00.308305 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.306413 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9"] Oct 30 22:30:00.311719 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.311688 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg"] Oct 30 22:30:00.319852 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.319818 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k"] Oct 30 22:30:00.364966 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.364922 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5"] Oct 30 22:30:00.370064 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.370029 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9"] Oct 30 22:30:00.374679 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.374641 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4vvk\" (UniqueName: \"kubernetes.io/projected/fece6ef8-ca00-4666-9860-0383ee19082c-kube-api-access-d4vvk\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5\" (UID: \"fece6ef8-ca00-4666-9860-0383ee19082c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5" Oct 30 22:30:00.374835 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.374751 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mln7n\" (UniqueName: \"kubernetes.io/projected/dcc289f1-3dd7-4182-bc07-8ca05b24b13b-kube-api-access-mln7n\") pod \"maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5\" (UID: \"dcc289f1-3dd7-4182-bc07-8ca05b24b13b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5" Oct 30 22:30:00.374835 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.374795 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcc289f1-3dd7-4182-bc07-8ca05b24b13b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5\" (UID: \"dcc289f1-3dd7-4182-bc07-8ca05b24b13b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5" Oct 30 22:30:00.374923 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.374838 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5vbz\" (UniqueName: \"kubernetes.io/projected/34557f8d-a341-4f36-9914-8ec5a9af1228-kube-api-access-m5vbz\") pod \"maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q\" (UID: \"34557f8d-a341-4f36-9914-8ec5a9af1228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q" Oct 30 22:30:00.374923 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.374871 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34557f8d-a341-4f36-9914-8ec5a9af1228-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q\" (UID: \"34557f8d-a341-4f36-9914-8ec5a9af1228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q" Oct 30 22:30:00.374985 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.374947 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8mcm\" (UniqueName: \"kubernetes.io/projected/cf68b963-460e-4af7-8f71-d00b8e0449fc-kube-api-access-b8mcm\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9\" (UID: \"cf68b963-460e-4af7-8f71-d00b8e0449fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9" Oct 30 22:30:00.375019 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.375004 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlgnn\" (UniqueName: \"kubernetes.io/projected/94535d2d-703e-4804-992a-474908adf7f2-kube-api-access-vlgnn\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs\" (UID: \"94535d2d-703e-4804-992a-474908adf7f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs" Oct 30 22:30:00.375053 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.375030 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc92x\" (UniqueName: \"kubernetes.io/projected/92efe8ed-fd67-4f56-a4d0-a7ef63e8964f-kube-api-access-bc92x\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k\" (UID: \"92efe8ed-fd67-4f56-a4d0-a7ef63e8964f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k" Oct 30 22:30:00.375094 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.375056 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9284e65-6486-487b-a207-a1b48b7ee19b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r\" (UID: \"d9284e65-6486-487b-a207-a1b48b7ee19b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r" Oct 30 22:30:00.375094 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.375081 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fece6ef8-ca00-4666-9860-0383ee19082c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5\" (UID: \"fece6ef8-ca00-4666-9860-0383ee19082c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5" Oct 30 22:30:00.375157 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.375110 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km2c2\" (UniqueName: \"kubernetes.io/projected/08446265-b751-4ff4-a788-488684eba3ea-kube-api-access-km2c2\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9\" (UID: \"08446265-b751-4ff4-a788-488684eba3ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9" Oct 30 22:30:00.375157 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.375139 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94535d2d-703e-4804-992a-474908adf7f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs\" (UID: \"94535d2d-703e-4804-992a-474908adf7f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs" Oct 30 22:30:00.375227 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.375190 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08446265-b751-4ff4-a788-488684eba3ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9\" (UID: \"08446265-b751-4ff4-a788-488684eba3ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9" Oct 30 22:30:00.375342 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.375319 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92efe8ed-fd67-4f56-a4d0-a7ef63e8964f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k\" (UID: \"92efe8ed-fd67-4f56-a4d0-a7ef63e8964f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k" Oct 30 22:30:00.375430 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.375356 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slpbr\" (UniqueName: \"kubernetes.io/projected/cd8cd9b2-105d-4115-a810-44e907efaba4-kube-api-access-slpbr\") pod \"maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg\" (UID: \"cd8cd9b2-105d-4115-a810-44e907efaba4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg" Oct 30 22:30:00.375430 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.375403 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p2sl\" (UniqueName: \"kubernetes.io/projected/d9284e65-6486-487b-a207-a1b48b7ee19b-kube-api-access-7p2sl\") pod \"maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r\" (UID: \"d9284e65-6486-487b-a207-a1b48b7ee19b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r" Oct 30 22:30:00.375545 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.375441 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8cd9b2-105d-4115-a810-44e907efaba4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg\" (UID: \"cd8cd9b2-105d-4115-a810-44e907efaba4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg" Oct 30 22:30:00.375545 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.375478 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf68b963-460e-4af7-8f71-d00b8e0449fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9\" (UID: \"cf68b963-460e-4af7-8f71-d00b8e0449fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9" Oct 30 22:30:00.376757 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.376730 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcc289f1-3dd7-4182-bc07-8ca05b24b13b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5\" (UID: \"dcc289f1-3dd7-4182-bc07-8ca05b24b13b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5" Oct 30 22:30:00.377453 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.377429 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r"] Oct 30 22:30:00.377560 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.377483 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5"] Oct 30 22:30:00.377740 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.377715 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34557f8d-a341-4f36-9914-8ec5a9af1228-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q\" (UID: \"34557f8d-a341-4f36-9914-8ec5a9af1228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q" Oct 30 22:30:00.378414 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.378390 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94535d2d-703e-4804-992a-474908adf7f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs\" (UID: \"94535d2d-703e-4804-992a-474908adf7f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs" Oct 30 22:30:00.378517 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.378456 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08446265-b751-4ff4-a788-488684eba3ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9\" (UID: \"08446265-b751-4ff4-a788-488684eba3ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9" Oct 30 22:30:00.379070 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.379045 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8cd9b2-105d-4115-a810-44e907efaba4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg\" (UID: \"cd8cd9b2-105d-4115-a810-44e907efaba4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg" Oct 30 22:30:00.433948 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.433878 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5vbz\" (UniqueName: \"kubernetes.io/projected/34557f8d-a341-4f36-9914-8ec5a9af1228-kube-api-access-m5vbz\") pod \"maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q\" (UID: \"34557f8d-a341-4f36-9914-8ec5a9af1228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q" Oct 30 22:30:00.433948 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.433876 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mln7n\" (UniqueName: \"kubernetes.io/projected/dcc289f1-3dd7-4182-bc07-8ca05b24b13b-kube-api-access-mln7n\") pod \"maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5\" (UID: \"dcc289f1-3dd7-4182-bc07-8ca05b24b13b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5" Oct 30 22:30:00.435180 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.435161 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlgnn\" (UniqueName: \"kubernetes.io/projected/94535d2d-703e-4804-992a-474908adf7f2-kube-api-access-vlgnn\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs\" (UID: \"94535d2d-703e-4804-992a-474908adf7f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs" Oct 30 22:30:00.435319 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.435301 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km2c2\" (UniqueName: \"kubernetes.io/projected/08446265-b751-4ff4-a788-488684eba3ea-kube-api-access-km2c2\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9\" (UID: \"08446265-b751-4ff4-a788-488684eba3ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9" Oct 30 22:30:00.464578 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.464545 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slpbr\" (UniqueName: \"kubernetes.io/projected/cd8cd9b2-105d-4115-a810-44e907efaba4-kube-api-access-slpbr\") pod \"maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg\" (UID: \"cd8cd9b2-105d-4115-a810-44e907efaba4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg" Oct 30 22:30:00.476567 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.476532 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92efe8ed-fd67-4f56-a4d0-a7ef63e8964f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k\" (UID: \"92efe8ed-fd67-4f56-a4d0-a7ef63e8964f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k" Oct 30 22:30:00.476731 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.476588 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p2sl\" (UniqueName: \"kubernetes.io/projected/d9284e65-6486-487b-a207-a1b48b7ee19b-kube-api-access-7p2sl\") pod \"maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r\" (UID: \"d9284e65-6486-487b-a207-a1b48b7ee19b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r" Oct 30 22:30:00.476731 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.476633 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf68b963-460e-4af7-8f71-d00b8e0449fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9\" (UID: \"cf68b963-460e-4af7-8f71-d00b8e0449fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9" Oct 30 22:30:00.476731 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.476695 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4vvk\" (UniqueName: \"kubernetes.io/projected/fece6ef8-ca00-4666-9860-0383ee19082c-kube-api-access-d4vvk\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5\" (UID: \"fece6ef8-ca00-4666-9860-0383ee19082c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5" Oct 30 22:30:00.476863 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.476812 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8mcm\" (UniqueName: \"kubernetes.io/projected/cf68b963-460e-4af7-8f71-d00b8e0449fc-kube-api-access-b8mcm\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9\" (UID: \"cf68b963-460e-4af7-8f71-d00b8e0449fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9" Oct 30 22:30:00.476923 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.476865 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc92x\" (UniqueName: \"kubernetes.io/projected/92efe8ed-fd67-4f56-a4d0-a7ef63e8964f-kube-api-access-bc92x\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k\" (UID: \"92efe8ed-fd67-4f56-a4d0-a7ef63e8964f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k" Oct 30 22:30:00.476923 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.476891 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9284e65-6486-487b-a207-a1b48b7ee19b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r\" (UID: \"d9284e65-6486-487b-a207-a1b48b7ee19b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r" Oct 30 22:30:00.476923 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.476918 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fece6ef8-ca00-4666-9860-0383ee19082c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5\" (UID: \"fece6ef8-ca00-4666-9860-0383ee19082c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5" Oct 30 22:30:00.477457 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.477437 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf68b963-460e-4af7-8f71-d00b8e0449fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9\" (UID: \"cf68b963-460e-4af7-8f71-d00b8e0449fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9" Oct 30 22:30:00.477540 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.477461 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92efe8ed-fd67-4f56-a4d0-a7ef63e8964f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k\" (UID: \"92efe8ed-fd67-4f56-a4d0-a7ef63e8964f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k" Oct 30 22:30:00.477582 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.477555 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9284e65-6486-487b-a207-a1b48b7ee19b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r\" (UID: \"d9284e65-6486-487b-a207-a1b48b7ee19b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r" Oct 30 22:30:00.477693 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.477676 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fece6ef8-ca00-4666-9860-0383ee19082c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5\" (UID: \"fece6ef8-ca00-4666-9860-0383ee19082c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5" Oct 30 22:30:00.504445 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.504403 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs" Oct 30 22:30:00.516648 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.516616 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q" Oct 30 22:30:00.525152 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.525117 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9" Oct 30 22:30:00.539322 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.539059 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc92x\" (UniqueName: \"kubernetes.io/projected/92efe8ed-fd67-4f56-a4d0-a7ef63e8964f-kube-api-access-bc92x\") pod \"maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k\" (UID: \"92efe8ed-fd67-4f56-a4d0-a7ef63e8964f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k" Oct 30 22:30:00.539739 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.539709 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p2sl\" (UniqueName: \"kubernetes.io/projected/d9284e65-6486-487b-a207-a1b48b7ee19b-kube-api-access-7p2sl\") pod \"maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r\" (UID: \"d9284e65-6486-487b-a207-a1b48b7ee19b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r" Oct 30 22:30:00.540766 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.540367 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4vvk\" (UniqueName: \"kubernetes.io/projected/fece6ef8-ca00-4666-9860-0383ee19082c-kube-api-access-d4vvk\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5\" (UID: \"fece6ef8-ca00-4666-9860-0383ee19082c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5" Oct 30 22:30:00.543712 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.543658 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8mcm\" (UniqueName: \"kubernetes.io/projected/cf68b963-460e-4af7-8f71-d00b8e0449fc-kube-api-access-b8mcm\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9\" (UID: \"cf68b963-460e-4af7-8f71-d00b8e0449fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9" Oct 30 22:30:00.554242 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.553989 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg" Oct 30 22:30:00.570219 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.570036 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5" Oct 30 22:30:00.579959 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.579744 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r" Oct 30 22:30:00.597937 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.597217 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k" Oct 30 22:30:00.616127 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.615530 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5" Oct 30 22:30:00.626212 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:00.625832 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9" Oct 30 22:30:01.104412 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.104370 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs"] Oct 30 22:30:01.104655 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:30:01.104631 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94535d2d_703e_4804_992a_474908adf7f2.slice/crio-6b408e2f1a5a6cbdb46ab5ffd87c316486e6ba80e3981fdb7295dfef1443ce6c WatchSource:0}: Error finding container 6b408e2f1a5a6cbdb46ab5ffd87c316486e6ba80e3981fdb7295dfef1443ce6c: Status 404 returned error can't find the container with id 6b408e2f1a5a6cbdb46ab5ffd87c316486e6ba80e3981fdb7295dfef1443ce6c Oct 30 22:30:01.150768 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:30:01.150726 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf68b963_460e_4af7_8f71_d00b8e0449fc.slice/crio-f0f09e0b35600e0620ffcfec9d6698d732d6b0936690217c3be924e96d492236 WatchSource:0}: Error finding container f0f09e0b35600e0620ffcfec9d6698d732d6b0936690217c3be924e96d492236: Status 404 returned error can't find the container with id f0f09e0b35600e0620ffcfec9d6698d732d6b0936690217c3be924e96d492236 Oct 30 22:30:01.150768 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.150743 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9"] Oct 30 22:30:01.153462 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:30:01.153441 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34557f8d_a341_4f36_9914_8ec5a9af1228.slice/crio-1e7699c03393d4aed87660d523248b2cd105383ee41171f857b6fc9121f14490 WatchSource:0}: Error finding container 1e7699c03393d4aed87660d523248b2cd105383ee41171f857b6fc9121f14490: Status 404 returned error can't find the container with id 1e7699c03393d4aed87660d523248b2cd105383ee41171f857b6fc9121f14490 Oct 30 22:30:01.154632 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:30:01.154606 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08446265_b751_4ff4_a788_488684eba3ea.slice/crio-8f3c196011167a2be497fa53245208f0ac37ada7a181a49c5b43b81ae7fd6247 WatchSource:0}: Error finding container 8f3c196011167a2be497fa53245208f0ac37ada7a181a49c5b43b81ae7fd6247: Status 404 returned error can't find the container with id 8f3c196011167a2be497fa53245208f0ac37ada7a181a49c5b43b81ae7fd6247 Oct 30 22:30:01.154870 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.154852 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9"] Oct 30 22:30:01.158957 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.158932 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q"] Oct 30 22:30:01.199017 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.198985 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5"] Oct 30 22:30:01.204190 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.204150 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg"] Oct 30 22:30:01.208222 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:30:01.208193 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8cd9b2_105d_4115_a810_44e907efaba4.slice/crio-ced9b291eda7c1b2c8a76543f9578df6e3715edaac6d42ae825d16f5fa58ef8b WatchSource:0}: Error finding container ced9b291eda7c1b2c8a76543f9578df6e3715edaac6d42ae825d16f5fa58ef8b: Status 404 returned error can't find the container with id ced9b291eda7c1b2c8a76543f9578df6e3715edaac6d42ae825d16f5fa58ef8b Oct 30 22:30:01.217513 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.217484 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9" event={"ID":"08446265-b751-4ff4-a788-488684eba3ea","Type":"ContainerStarted","Data":"8f3c196011167a2be497fa53245208f0ac37ada7a181a49c5b43b81ae7fd6247"} Oct 30 22:30:01.218959 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.218928 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs" event={"ID":"94535d2d-703e-4804-992a-474908adf7f2","Type":"ContainerStarted","Data":"6b408e2f1a5a6cbdb46ab5ffd87c316486e6ba80e3981fdb7295dfef1443ce6c"} Oct 30 22:30:01.220413 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.220387 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9" event={"ID":"cf68b963-460e-4af7-8f71-d00b8e0449fc","Type":"ContainerStarted","Data":"f0f09e0b35600e0620ffcfec9d6698d732d6b0936690217c3be924e96d492236"} Oct 30 22:30:01.221683 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.221655 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q" event={"ID":"34557f8d-a341-4f36-9914-8ec5a9af1228","Type":"ContainerStarted","Data":"1e7699c03393d4aed87660d523248b2cd105383ee41171f857b6fc9121f14490"} Oct 30 22:30:01.260574 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.260544 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k"] Oct 30 22:30:01.268662 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.268635 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5"] Oct 30 22:30:01.290882 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.290852 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r"] Oct 30 22:30:01.303852 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:30:01.303810 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92efe8ed_fd67_4f56_a4d0_a7ef63e8964f.slice/crio-edbcd2e167fa4ee3da44a446ccb3171b7e047d5871d43a2ad570beb271f51c3e WatchSource:0}: Error finding container edbcd2e167fa4ee3da44a446ccb3171b7e047d5871d43a2ad570beb271f51c3e: Status 404 returned error can't find the container with id edbcd2e167fa4ee3da44a446ccb3171b7e047d5871d43a2ad570beb271f51c3e Oct 30 22:30:01.402114 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:30:01.402029 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfece6ef8_ca00_4666_9860_0383ee19082c.slice/crio-f01dbd394e478dea84d369ed6b5a01b24b58c853b18d234bd3d946c4fac0b9a5 WatchSource:0}: Error finding container f01dbd394e478dea84d369ed6b5a01b24b58c853b18d234bd3d946c4fac0b9a5: Status 404 returned error can't find the container with id f01dbd394e478dea84d369ed6b5a01b24b58c853b18d234bd3d946c4fac0b9a5 Oct 30 22:30:01.403238 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:30:01.403156 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9284e65_6486_487b_a207_a1b48b7ee19b.slice/crio-4852be01ac0bcdbd841cd8cd6c8daac5b09b3d00c0895bac35b86967c70990fe WatchSource:0}: Error finding container 4852be01ac0bcdbd841cd8cd6c8daac5b09b3d00c0895bac35b86967c70990fe: Status 404 returned error can't find the container with id 4852be01ac0bcdbd841cd8cd6c8daac5b09b3d00c0895bac35b86967c70990fe Oct 30 22:30:01.650502 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.650451 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:30:01.699091 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.698986 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-credential-socket\") pod \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " Oct 30 22:30:01.699908 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.699821 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-data" pod "ab56f9ed-7e2f-4a74-8dda-53df36c32f23" (UID: "ab56f9ed-7e2f-4a74-8dda-53df36c32f23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:01.700102 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.700079 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-data\") pod \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " Oct 30 22:30:01.700495 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.700477 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-envoy\") pod \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " Oct 30 22:30:01.701123 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.701096 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-workload-socket\") pod \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " Oct 30 22:30:01.701300 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.701159 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-podinfo\") pod \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " Oct 30 22:30:01.701300 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.701193 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-token\" (UniqueName: \"kubernetes.io/projected/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-token\") pod \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " Oct 30 22:30:01.701300 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.701265 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istiod-ca-cert\") pod \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " Oct 30 22:30:01.701498 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.701344 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-credential-socket" pod "ab56f9ed-7e2f-4a74-8dda-53df36c32f23" (UID: "ab56f9ed-7e2f-4a74-8dda-53df36c32f23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:01.701761 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.701741 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-workload-socket" pod "ab56f9ed-7e2f-4a74-8dda-53df36c32f23" (UID: "ab56f9ed-7e2f-4a74-8dda-53df36c32f23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:01.702604 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.702580 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istiod-ca-cert" pod "ab56f9ed-7e2f-4a74-8dda-53df36c32f23" (UID: "ab56f9ed-7e2f-4a74-8dda-53df36c32f23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:01.702748 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.702731 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-workload-certs\") pod \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " Oct 30 22:30:01.702837 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.702808 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-kube-api-access-ml8lm\" (UniqueName: \"kubernetes.io/projected/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-kube-api-access-ml8lm\") pod \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\" (UID: \"ab56f9ed-7e2f-4a74-8dda-53df36c32f23\") " Oct 30 22:30:01.703201 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.703182 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-workload-certs" pod "ab56f9ed-7e2f-4a74-8dda-53df36c32f23" (UID: "ab56f9ed-7e2f-4a74-8dda-53df36c32f23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:01.703516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.703496 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:01.703671 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.703646 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:01.703740 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.703679 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:01.703740 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.703695 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:01.703740 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.703710 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:01.706489 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.706459 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-envoy" pod "ab56f9ed-7e2f-4a74-8dda-53df36c32f23" (UID: "ab56f9ed-7e2f-4a74-8dda-53df36c32f23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:01.706703 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.706662 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-podinfo" pod "ab56f9ed-7e2f-4a74-8dda-53df36c32f23" (UID: "ab56f9ed-7e2f-4a74-8dda-53df36c32f23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:30:01.706703 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.706677 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-token" pod "ab56f9ed-7e2f-4a74-8dda-53df36c32f23" (UID: "ab56f9ed-7e2f-4a74-8dda-53df36c32f23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:01.707861 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.707835 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-kube-api-access-ml8lm" pod "ab56f9ed-7e2f-4a74-8dda-53df36c32f23" (UID: "ab56f9ed-7e2f-4a74-8dda-53df36c32f23"). InnerVolumeSpecName "kube-api-access-ml8lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:01.756292 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.756234 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:30:01.804200 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.804162 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-kube-api-access-hhphr\" (UniqueName: \"kubernetes.io/projected/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-kube-api-access-hhphr\") pod \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " Oct 30 22:30:01.804367 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.804328 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istiod-ca-cert\") pod \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " Oct 30 22:30:01.804483 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.804468 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-token\" (UniqueName: \"kubernetes.io/projected/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-token\") pod \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " Oct 30 22:30:01.804530 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.804517 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-podinfo\") pod \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " Oct 30 22:30:01.804579 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.804568 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-data\") pod \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " Oct 30 22:30:01.804635 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.804606 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-credential-socket\") pod \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " Oct 30 22:30:01.804635 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.804628 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-envoy\") pod \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " Oct 30 22:30:01.804737 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.804666 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-workload-socket\") pod \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " Oct 30 22:30:01.804737 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.804706 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-workload-certs\") pod \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\" (UID: \"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3\") " Oct 30 22:30:01.805187 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.805166 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:01.805187 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.805187 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:01.806318 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.805196 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:01.806318 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.805206 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml8lm\" (UniqueName: \"kubernetes.io/projected/ab56f9ed-7e2f-4a74-8dda-53df36c32f23-kube-api-access-ml8lm\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:01.806318 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.805492 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-workload-certs" pod "f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3" (UID: "f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:01.806318 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.805793 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-credential-socket" pod "f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3" (UID: "f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:01.806318 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.805827 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istiod-ca-cert" pod "f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3" (UID: "f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:01.806318 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.806005 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-data" pod "f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3" (UID: "f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:01.806318 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.806058 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-workload-socket" pod "f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3" (UID: "f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:01.807172 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.807142 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-kube-api-access-hhphr" pod "f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3" (UID: "f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3"). InnerVolumeSpecName "kube-api-access-hhphr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:01.807326 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.807300 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-token" pod "f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3" (UID: "f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:01.807738 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.807715 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-envoy" pod "f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3" (UID: "f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:01.809196 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.809175 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-podinfo" pod "f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3" (UID: "f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:30:01.906507 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.906463 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:01.906507 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.906508 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhphr\" (UniqueName: \"kubernetes.io/projected/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-kube-api-access-hhphr\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:01.906719 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.906523 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:01.906719 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.906536 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:01.906719 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.906551 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:01.906719 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.906564 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:01.906719 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.906579 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:01.906719 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.906592 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:01.906719 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:01.906606 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:02.232622 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.232580 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg" event={"ID":"cd8cd9b2-105d-4115-a810-44e907efaba4","Type":"ContainerStarted","Data":"a884347d6606136dd2a5e2d21ed4b7935ba30f96e0af08c0236829af93b77e5f"} Oct 30 22:30:02.233102 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.232647 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg" event={"ID":"cd8cd9b2-105d-4115-a810-44e907efaba4","Type":"ContainerStarted","Data":"ced9b291eda7c1b2c8a76543f9578df6e3715edaac6d42ae825d16f5fa58ef8b"} Oct 30 22:30:02.234378 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.234348 6006 generic.go:356] "Generic (PLEG): container finished" podID="cf68b963-460e-4af7-8f71-d00b8e0449fc" containerID="a664201cae63ecbcf275c6385eeb7766f004b5dc88f901f07bc06139850adf6f" exitCode=60 Oct 30 22:30:02.234490 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.234386 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9" event={"ID":"cf68b963-460e-4af7-8f71-d00b8e0449fc","Type":"ContainerDied","Data":"a664201cae63ecbcf275c6385eeb7766f004b5dc88f901f07bc06139850adf6f"} Oct 30 22:30:02.234726 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.234710 6006 scope.go:122] "RemoveContainer" containerID="a664201cae63ecbcf275c6385eeb7766f004b5dc88f901f07bc06139850adf6f" Oct 30 22:30:02.236327 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.236301 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r" event={"ID":"d9284e65-6486-487b-a207-a1b48b7ee19b","Type":"ContainerStarted","Data":"fb3c0dd6f149cc322de3f89088d8411bc1ec1cda9b2954662772cedc78a208fb"} Oct 30 22:30:02.236454 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.236348 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r" event={"ID":"d9284e65-6486-487b-a207-a1b48b7ee19b","Type":"ContainerStarted","Data":"4852be01ac0bcdbd841cd8cd6c8daac5b09b3d00c0895bac35b86967c70990fe"} Oct 30 22:30:02.238168 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.238065 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q" event={"ID":"34557f8d-a341-4f36-9914-8ec5a9af1228","Type":"ContainerStarted","Data":"e05e596000bf50155a63b74dafe935a001f34a2fc2e77b81a0f65c7447ad5d00"} Oct 30 22:30:02.239574 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.239548 6006 generic.go:356] "Generic (PLEG): container finished" podID="ab56f9ed-7e2f-4a74-8dda-53df36c32f23" containerID="a4885693acc976444deef6cd022219f61bd92b56374102a38907c95f2a0502a1" exitCode=0 Oct 30 22:30:02.239681 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.239607 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" Oct 30 22:30:02.239681 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.239627 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" event={"ID":"ab56f9ed-7e2f-4a74-8dda-53df36c32f23","Type":"ContainerDied","Data":"a4885693acc976444deef6cd022219f61bd92b56374102a38907c95f2a0502a1"} Oct 30 22:30:02.239681 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.239668 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9" event={"ID":"ab56f9ed-7e2f-4a74-8dda-53df36c32f23","Type":"ContainerDied","Data":"2c6454804872d766de22f5a2c3c59cc538051be1a76df198942c134701152f8e"} Oct 30 22:30:02.239847 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.239697 6006 scope.go:122] "RemoveContainer" containerID="a4885693acc976444deef6cd022219f61bd92b56374102a38907c95f2a0502a1" Oct 30 22:30:02.241579 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.241554 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5" event={"ID":"fece6ef8-ca00-4666-9860-0383ee19082c","Type":"ContainerStarted","Data":"73f6654c6cb541a2b9e2b81e2b1df42648d9a59745541546c89385ae4fb33c10"} Oct 30 22:30:02.241695 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.241603 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5" event={"ID":"fece6ef8-ca00-4666-9860-0383ee19082c","Type":"ContainerStarted","Data":"f01dbd394e478dea84d369ed6b5a01b24b58c853b18d234bd3d946c4fac0b9a5"} Oct 30 22:30:02.243177 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.243152 6006 generic.go:356] "Generic (PLEG): container finished" podID="f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3" containerID="009be692572310e3168485877a4133d7ffcab468040cfda7549f62534e356956" exitCode=0 Oct 30 22:30:02.243308 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.243196 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" Oct 30 22:30:02.243372 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.243220 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" event={"ID":"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3","Type":"ContainerDied","Data":"009be692572310e3168485877a4133d7ffcab468040cfda7549f62534e356956"} Oct 30 22:30:02.243441 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.243398 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p" event={"ID":"f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3","Type":"ContainerDied","Data":"496df27568e514d6f03ac0ba1694637ff1e0ad4ec1acd5c626202e691bcef762"} Oct 30 22:30:02.245213 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.245191 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k" event={"ID":"92efe8ed-fd67-4f56-a4d0-a7ef63e8964f","Type":"ContainerStarted","Data":"87894f30b5c647fa35b16467bf326ec1c6fa17880e9970e1d4eede5e725fcdae"} Oct 30 22:30:02.245352 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.245238 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k" event={"ID":"92efe8ed-fd67-4f56-a4d0-a7ef63e8964f","Type":"ContainerStarted","Data":"edbcd2e167fa4ee3da44a446ccb3171b7e047d5871d43a2ad570beb271f51c3e"} Oct 30 22:30:02.247080 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.247058 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5" event={"ID":"dcc289f1-3dd7-4182-bc07-8ca05b24b13b","Type":"ContainerStarted","Data":"20ea3d5fd09f9ca3345aa72d88c7c13b5fb899e38bc529c0dca73e066305e2e9"} Oct 30 22:30:02.247174 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.247110 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5" event={"ID":"dcc289f1-3dd7-4182-bc07-8ca05b24b13b","Type":"ContainerStarted","Data":"858fa0a942aefc25aaecf608c056588d86a70d5267671591ef3a2edd96f87515"} Oct 30 22:30:02.248877 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.248841 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9" event={"ID":"08446265-b751-4ff4-a788-488684eba3ea","Type":"ContainerStarted","Data":"7cac426c5d1a3017c40c2fcec6db79be493ad010edd65202c71beebf1b73d9e8"} Oct 30 22:30:02.250850 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.250829 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs" event={"ID":"94535d2d-703e-4804-992a-474908adf7f2","Type":"ContainerStarted","Data":"0e312043a1734ab3c101241289e5ae6ea691dbfe4835f97c3f71c61920a14633"} Oct 30 22:30:02.265398 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.265370 6006 scope.go:122] "RemoveContainer" containerID="a4885693acc976444deef6cd022219f61bd92b56374102a38907c95f2a0502a1" Oct 30 22:30:02.265816 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:02.265787 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4885693acc976444deef6cd022219f61bd92b56374102a38907c95f2a0502a1\": container with ID starting with a4885693acc976444deef6cd022219f61bd92b56374102a38907c95f2a0502a1 not found: ID does not exist" containerID="a4885693acc976444deef6cd022219f61bd92b56374102a38907c95f2a0502a1" Oct 30 22:30:02.265900 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.265829 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4885693acc976444deef6cd022219f61bd92b56374102a38907c95f2a0502a1"} err="failed to get container status \"a4885693acc976444deef6cd022219f61bd92b56374102a38907c95f2a0502a1\": rpc error: code = NotFound desc = could not find container \"a4885693acc976444deef6cd022219f61bd92b56374102a38907c95f2a0502a1\": container with ID starting with a4885693acc976444deef6cd022219f61bd92b56374102a38907c95f2a0502a1 not found: ID does not exist" Oct 30 22:30:02.265900 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.265855 6006 scope.go:122] "RemoveContainer" containerID="009be692572310e3168485877a4133d7ffcab468040cfda7549f62534e356956" Oct 30 22:30:02.283222 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.283198 6006 scope.go:122] "RemoveContainer" containerID="009be692572310e3168485877a4133d7ffcab468040cfda7549f62534e356956" Oct 30 22:30:02.283617 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:02.283595 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"009be692572310e3168485877a4133d7ffcab468040cfda7549f62534e356956\": container with ID starting with 009be692572310e3168485877a4133d7ffcab468040cfda7549f62534e356956 not found: ID does not exist" containerID="009be692572310e3168485877a4133d7ffcab468040cfda7549f62534e356956" Oct 30 22:30:02.283715 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.283624 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"009be692572310e3168485877a4133d7ffcab468040cfda7549f62534e356956"} err="failed to get container status \"009be692572310e3168485877a4133d7ffcab468040cfda7549f62534e356956\": rpc error: code = NotFound desc = could not find container \"009be692572310e3168485877a4133d7ffcab468040cfda7549f62534e356956\": container with ID starting with 009be692572310e3168485877a4133d7ffcab468040cfda7549f62534e356956 not found: ID does not exist" Oct 30 22:30:02.344810 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.344748 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg" podStartSLOduration=2.344727207 podStartE2EDuration="2.344727207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.341965092 +0000 UTC m=+2404.894141393" watchObservedRunningTime="2026-10-30 22:30:02.344727207 +0000 UTC m=+2404.896903507" Oct 30 22:30:02.457005 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.456938 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r" podStartSLOduration=2.456919048 podStartE2EDuration="2.456919048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.454093936 +0000 UTC m=+2405.006270237" watchObservedRunningTime="2026-10-30 22:30:02.456919048 +0000 UTC m=+2405.009095353" Oct 30 22:30:02.577172 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.569144 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k" podStartSLOduration=2.5691260590000002 podStartE2EDuration="2.569126059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.568666078 +0000 UTC m=+2405.120842380" watchObservedRunningTime="2026-10-30 22:30:02.569126059 +0000 UTC m=+2405.121302361" Oct 30 22:30:02.677223 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.677179 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9"] Oct 30 22:30:02.681064 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.681024 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-65e59f-openshift-default-596d74bb4f-kj7n9"] Oct 30 22:30:02.783581 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.783433 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p"] Oct 30 22:30:02.788823 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.788791 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bc714d-openshift-default-67558b95cf-ph28p"] Oct 30 22:30:02.907054 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:02.907023 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:30:03.006590 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.006415 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5" podStartSLOduration=3.006395391 podStartE2EDuration="3.006395391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.003285089 +0000 UTC m=+2405.555461389" watchObservedRunningTime="2026-10-30 22:30:03.006395391 +0000 UTC m=+2405.558571693" Oct 30 22:30:03.025349 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.025208 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-istio-envoy\") pod \"f33ebace-e652-4f07-bbce-9c334faa3948\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " Oct 30 22:30:03.025349 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.025297 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-istio-data\") pod \"f33ebace-e652-4f07-bbce-9c334faa3948\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " Oct 30 22:30:03.025349 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.025348 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-workload-certs\") pod \"f33ebace-e652-4f07-bbce-9c334faa3948\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " Oct 30 22:30:03.025658 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.025391 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f33ebace-e652-4f07-bbce-9c334faa3948-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f33ebace-e652-4f07-bbce-9c334faa3948-istiod-ca-cert\") pod \"f33ebace-e652-4f07-bbce-9c334faa3948\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " Oct 30 22:30:03.025658 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.025495 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f33ebace-e652-4f07-bbce-9c334faa3948-kube-api-access-ct25w\" (UniqueName: \"kubernetes.io/projected/f33ebace-e652-4f07-bbce-9c334faa3948-kube-api-access-ct25w\") pod \"f33ebace-e652-4f07-bbce-9c334faa3948\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " Oct 30 22:30:03.025658 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.025543 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f33ebace-e652-4f07-bbce-9c334faa3948-istio-token\" (UniqueName: \"kubernetes.io/projected/f33ebace-e652-4f07-bbce-9c334faa3948-istio-token\") pod \"f33ebace-e652-4f07-bbce-9c334faa3948\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " Oct 30 22:30:03.025658 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.025575 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-istio-data" pod "f33ebace-e652-4f07-bbce-9c334faa3948" (UID: "f33ebace-e652-4f07-bbce-9c334faa3948"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:03.025658 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.025614 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f33ebace-e652-4f07-bbce-9c334faa3948-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f33ebace-e652-4f07-bbce-9c334faa3948-istio-podinfo\") pod \"f33ebace-e652-4f07-bbce-9c334faa3948\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " Oct 30 22:30:03.025901 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.025728 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-credential-socket\") pod \"f33ebace-e652-4f07-bbce-9c334faa3948\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " Oct 30 22:30:03.025901 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.025762 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-workload-socket\") pod \"f33ebace-e652-4f07-bbce-9c334faa3948\" (UID: \"f33ebace-e652-4f07-bbce-9c334faa3948\") " Oct 30 22:30:03.025901 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.025801 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-workload-certs" pod "f33ebace-e652-4f07-bbce-9c334faa3948" (UID: "f33ebace-e652-4f07-bbce-9c334faa3948"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:03.026289 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.026244 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:03.026387 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.026292 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:03.026544 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.026518 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f33ebace-e652-4f07-bbce-9c334faa3948-istiod-ca-cert" pod "f33ebace-e652-4f07-bbce-9c334faa3948" (UID: "f33ebace-e652-4f07-bbce-9c334faa3948"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:03.026970 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.026946 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-credential-socket" pod "f33ebace-e652-4f07-bbce-9c334faa3948" (UID: "f33ebace-e652-4f07-bbce-9c334faa3948"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:03.027068 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.027002 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-workload-socket" pod "f33ebace-e652-4f07-bbce-9c334faa3948" (UID: "f33ebace-e652-4f07-bbce-9c334faa3948"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:03.027996 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.027972 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-istio-envoy" pod "f33ebace-e652-4f07-bbce-9c334faa3948" (UID: "f33ebace-e652-4f07-bbce-9c334faa3948"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:03.028381 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.028357 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f33ebace-e652-4f07-bbce-9c334faa3948-istio-podinfo" pod "f33ebace-e652-4f07-bbce-9c334faa3948" (UID: "f33ebace-e652-4f07-bbce-9c334faa3948"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:30:03.028535 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.028517 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f33ebace-e652-4f07-bbce-9c334faa3948-istio-token" pod "f33ebace-e652-4f07-bbce-9c334faa3948" (UID: "f33ebace-e652-4f07-bbce-9c334faa3948"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:03.028874 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.028860 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f33ebace-e652-4f07-bbce-9c334faa3948-kube-api-access-ct25w" pod "f33ebace-e652-4f07-bbce-9c334faa3948" (UID: "f33ebace-e652-4f07-bbce-9c334faa3948"). InnerVolumeSpecName "kube-api-access-ct25w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:03.118183 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.118112 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs" podStartSLOduration=3.118095655 podStartE2EDuration="3.118095655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.114343127 +0000 UTC m=+2405.666519429" watchObservedRunningTime="2026-10-30 22:30:03.118095655 +0000 UTC m=+2405.670271957" Oct 30 22:30:03.127431 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.127396 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:03.127431 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.127432 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f33ebace-e652-4f07-bbce-9c334faa3948-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:03.127675 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.127448 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct25w\" (UniqueName: \"kubernetes.io/projected/f33ebace-e652-4f07-bbce-9c334faa3948-kube-api-access-ct25w\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:03.127675 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.127463 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f33ebace-e652-4f07-bbce-9c334faa3948-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:03.127675 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.127476 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f33ebace-e652-4f07-bbce-9c334faa3948-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:03.127675 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.127490 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:03.127675 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.127504 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33ebace-e652-4f07-bbce-9c334faa3948-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:03.225635 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.225563 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9" podStartSLOduration=3.225542733 podStartE2EDuration="3.225542733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.223169855 +0000 UTC m=+2405.775346154" watchObservedRunningTime="2026-10-30 22:30:03.225542733 +0000 UTC m=+2405.777719027" Oct 30 22:30:03.260700 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.260664 6006 generic.go:356] "Generic (PLEG): container finished" podID="d9284e65-6486-487b-a207-a1b48b7ee19b" containerID="fb3c0dd6f149cc322de3f89088d8411bc1ec1cda9b2954662772cedc78a208fb" exitCode=60 Oct 30 22:30:03.261197 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.260750 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r" event={"ID":"d9284e65-6486-487b-a207-a1b48b7ee19b","Type":"ContainerDied","Data":"fb3c0dd6f149cc322de3f89088d8411bc1ec1cda9b2954662772cedc78a208fb"} Oct 30 22:30:03.261197 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.261081 6006 scope.go:122] "RemoveContainer" containerID="fb3c0dd6f149cc322de3f89088d8411bc1ec1cda9b2954662772cedc78a208fb" Oct 30 22:30:03.262625 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.262600 6006 generic.go:356] "Generic (PLEG): container finished" podID="34557f8d-a341-4f36-9914-8ec5a9af1228" containerID="e05e596000bf50155a63b74dafe935a001f34a2fc2e77b81a0f65c7447ad5d00" exitCode=60 Oct 30 22:30:03.262741 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.262687 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q" event={"ID":"34557f8d-a341-4f36-9914-8ec5a9af1228","Type":"ContainerDied","Data":"e05e596000bf50155a63b74dafe935a001f34a2fc2e77b81a0f65c7447ad5d00"} Oct 30 22:30:03.262961 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.262938 6006 scope.go:122] "RemoveContainer" containerID="e05e596000bf50155a63b74dafe935a001f34a2fc2e77b81a0f65c7447ad5d00" Oct 30 22:30:03.265665 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.265640 6006 generic.go:356] "Generic (PLEG): container finished" podID="fece6ef8-ca00-4666-9860-0383ee19082c" containerID="73f6654c6cb541a2b9e2b81e2b1df42648d9a59745541546c89385ae4fb33c10" exitCode=60 Oct 30 22:30:03.265765 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.265723 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5" event={"ID":"fece6ef8-ca00-4666-9860-0383ee19082c","Type":"ContainerDied","Data":"73f6654c6cb541a2b9e2b81e2b1df42648d9a59745541546c89385ae4fb33c10"} Oct 30 22:30:03.265913 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.265897 6006 scope.go:122] "RemoveContainer" containerID="73f6654c6cb541a2b9e2b81e2b1df42648d9a59745541546c89385ae4fb33c10" Oct 30 22:30:03.268557 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.268532 6006 generic.go:356] "Generic (PLEG): container finished" podID="92efe8ed-fd67-4f56-a4d0-a7ef63e8964f" containerID="87894f30b5c647fa35b16467bf326ec1c6fa17880e9970e1d4eede5e725fcdae" exitCode=60 Oct 30 22:30:03.268671 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.268613 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k" event={"ID":"92efe8ed-fd67-4f56-a4d0-a7ef63e8964f","Type":"ContainerDied","Data":"87894f30b5c647fa35b16467bf326ec1c6fa17880e9970e1d4eede5e725fcdae"} Oct 30 22:30:03.268811 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.268796 6006 scope.go:122] "RemoveContainer" containerID="87894f30b5c647fa35b16467bf326ec1c6fa17880e9970e1d4eede5e725fcdae" Oct 30 22:30:03.270276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.270240 6006 generic.go:356] "Generic (PLEG): container finished" podID="dcc289f1-3dd7-4182-bc07-8ca05b24b13b" containerID="20ea3d5fd09f9ca3345aa72d88c7c13b5fb899e38bc529c0dca73e066305e2e9" exitCode=60 Oct 30 22:30:03.270369 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.270288 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5" event={"ID":"dcc289f1-3dd7-4182-bc07-8ca05b24b13b","Type":"ContainerDied","Data":"20ea3d5fd09f9ca3345aa72d88c7c13b5fb899e38bc529c0dca73e066305e2e9"} Oct 30 22:30:03.270546 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.270528 6006 scope.go:122] "RemoveContainer" containerID="20ea3d5fd09f9ca3345aa72d88c7c13b5fb899e38bc529c0dca73e066305e2e9" Oct 30 22:30:03.272002 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.271979 6006 generic.go:356] "Generic (PLEG): container finished" podID="08446265-b751-4ff4-a788-488684eba3ea" containerID="7cac426c5d1a3017c40c2fcec6db79be493ad010edd65202c71beebf1b73d9e8" exitCode=60 Oct 30 22:30:03.272089 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.272004 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9" event={"ID":"08446265-b751-4ff4-a788-488684eba3ea","Type":"ContainerDied","Data":"7cac426c5d1a3017c40c2fcec6db79be493ad010edd65202c71beebf1b73d9e8"} Oct 30 22:30:03.272303 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.272285 6006 scope.go:122] "RemoveContainer" containerID="7cac426c5d1a3017c40c2fcec6db79be493ad010edd65202c71beebf1b73d9e8" Oct 30 22:30:03.276731 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.276708 6006 generic.go:356] "Generic (PLEG): container finished" podID="94535d2d-703e-4804-992a-474908adf7f2" containerID="0e312043a1734ab3c101241289e5ae6ea691dbfe4835f97c3f71c61920a14633" exitCode=60 Oct 30 22:30:03.276883 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.276784 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs" event={"ID":"94535d2d-703e-4804-992a-474908adf7f2","Type":"ContainerDied","Data":"0e312043a1734ab3c101241289e5ae6ea691dbfe4835f97c3f71c61920a14633"} Oct 30 22:30:03.276967 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.276949 6006 scope.go:122] "RemoveContainer" containerID="0e312043a1734ab3c101241289e5ae6ea691dbfe4835f97c3f71c61920a14633" Oct 30 22:30:03.278517 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.278495 6006 generic.go:356] "Generic (PLEG): container finished" podID="cd8cd9b2-105d-4115-a810-44e907efaba4" containerID="a884347d6606136dd2a5e2d21ed4b7935ba30f96e0af08c0236829af93b77e5f" exitCode=60 Oct 30 22:30:03.278594 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.278578 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg" event={"ID":"cd8cd9b2-105d-4115-a810-44e907efaba4","Type":"ContainerDied","Data":"a884347d6606136dd2a5e2d21ed4b7935ba30f96e0af08c0236829af93b77e5f"} Oct 30 22:30:03.278763 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.278749 6006 scope.go:122] "RemoveContainer" containerID="a884347d6606136dd2a5e2d21ed4b7935ba30f96e0af08c0236829af93b77e5f" Oct 30 22:30:03.285989 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.285943 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9" event={"ID":"cf68b963-460e-4af7-8f71-d00b8e0449fc","Type":"ContainerStarted","Data":"df287728602da41b86370f06fa955c5104ff0c6cb13ebc62ad59d73306e2ea3f"} Oct 30 22:30:03.287923 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.287901 6006 generic.go:356] "Generic (PLEG): container finished" podID="f33ebace-e652-4f07-bbce-9c334faa3948" containerID="a7f28a559a912c74ad8c03267b0ae6427e548a2e7c3a57364b48b76ce5dc6f12" exitCode=0 Oct 30 22:30:03.288027 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.287968 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" event={"ID":"f33ebace-e652-4f07-bbce-9c334faa3948","Type":"ContainerDied","Data":"a7f28a559a912c74ad8c03267b0ae6427e548a2e7c3a57364b48b76ce5dc6f12"} Oct 30 22:30:03.288027 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.288000 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" event={"ID":"f33ebace-e652-4f07-bbce-9c334faa3948","Type":"ContainerDied","Data":"8957b8c9d255a0621c22c8de0a48a021d4be9542753dcbea392ec1bc9643a59b"} Oct 30 22:30:03.288027 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.288023 6006 scope.go:122] "RemoveContainer" containerID="a7f28a559a912c74ad8c03267b0ae6427e548a2e7c3a57364b48b76ce5dc6f12" Oct 30 22:30:03.288185 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.288146 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9" Oct 30 22:30:03.335710 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.335651 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q" podStartSLOduration=3.335621261 podStartE2EDuration="3.335621261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.335049071 +0000 UTC m=+2405.887225371" watchObservedRunningTime="2026-10-30 22:30:03.335621261 +0000 UTC m=+2405.887797562" Oct 30 22:30:03.419220 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.419097 6006 scope.go:122] "RemoveContainer" containerID="a7f28a559a912c74ad8c03267b0ae6427e548a2e7c3a57364b48b76ce5dc6f12" Oct 30 22:30:03.419547 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:03.419513 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7f28a559a912c74ad8c03267b0ae6427e548a2e7c3a57364b48b76ce5dc6f12\": container with ID starting with a7f28a559a912c74ad8c03267b0ae6427e548a2e7c3a57364b48b76ce5dc6f12 not found: ID does not exist" containerID="a7f28a559a912c74ad8c03267b0ae6427e548a2e7c3a57364b48b76ce5dc6f12" Oct 30 22:30:03.419641 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.419561 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7f28a559a912c74ad8c03267b0ae6427e548a2e7c3a57364b48b76ce5dc6f12"} err="failed to get container status \"a7f28a559a912c74ad8c03267b0ae6427e548a2e7c3a57364b48b76ce5dc6f12\": rpc error: code = NotFound desc = could not find container \"a7f28a559a912c74ad8c03267b0ae6427e548a2e7c3a57364b48b76ce5dc6f12\": container with ID starting with a7f28a559a912c74ad8c03267b0ae6427e548a2e7c3a57364b48b76ce5dc6f12 not found: ID does not exist" Oct 30 22:30:03.446108 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.446053 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5" podStartSLOduration=3.446037106 podStartE2EDuration="3.446037106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.44365786 +0000 UTC m=+2405.995834160" watchObservedRunningTime="2026-10-30 22:30:03.446037106 +0000 UTC m=+2405.998213443" Oct 30 22:30:03.610640 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.610607 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab56f9ed-7e2f-4a74-8dda-53df36c32f23" path="/var/lib/kubelet/pods/ab56f9ed-7e2f-4a74-8dda-53df36c32f23/volumes" Oct 30 22:30:03.611583 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:03.611562 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3" path="/var/lib/kubelet/pods/f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3/volumes" Oct 30 22:30:04.304753 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:04.298457 6006 generic.go:356] "Generic (PLEG): container finished" podID="cf68b963-460e-4af7-8f71-d00b8e0449fc" containerID="df287728602da41b86370f06fa955c5104ff0c6cb13ebc62ad59d73306e2ea3f" exitCode=60 Oct 30 22:30:04.304753 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:04.300169 6006 scope.go:122] "RemoveContainer" containerID="df287728602da41b86370f06fa955c5104ff0c6cb13ebc62ad59d73306e2ea3f" Oct 30 22:30:04.304753 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:04.300606 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9_odh-ai-gateway-infra(cf68b963-460e-4af7-8f71-d00b8e0449fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9" podUID="cf68b963-460e-4af7-8f71-d00b8e0449fc" Oct 30 22:30:04.304753 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:04.300662 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9" event={"ID":"cf68b963-460e-4af7-8f71-d00b8e0449fc","Type":"ContainerDied","Data":"df287728602da41b86370f06fa955c5104ff0c6cb13ebc62ad59d73306e2ea3f"} Oct 30 22:30:04.304753 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:04.300710 6006 scope.go:122] "RemoveContainer" containerID="a664201cae63ecbcf275c6385eeb7766f004b5dc88f901f07bc06139850adf6f" Oct 30 22:30:04.312191 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:04.312163 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r" event={"ID":"d9284e65-6486-487b-a207-a1b48b7ee19b","Type":"ContainerStarted","Data":"1cb5b41eea798c8c6a7c0d03e9ca84f9c4b686f8564d6bcf9efea1550a3d7a41"} Oct 30 22:30:04.315969 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:04.315936 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q" event={"ID":"34557f8d-a341-4f36-9914-8ec5a9af1228","Type":"ContainerStarted","Data":"653f41141456ee5c763761b2caca573eaab8245be67e5a00297a5d67e4e7a0ab"} Oct 30 22:30:04.319601 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:04.319567 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k" event={"ID":"92efe8ed-fd67-4f56-a4d0-a7ef63e8964f","Type":"ContainerStarted","Data":"9c7add8c86095b54e435a13dca50fd7198a8a2642ead825a38ed925923aa4654"} Oct 30 22:30:04.322860 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:04.322828 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5" event={"ID":"dcc289f1-3dd7-4182-bc07-8ca05b24b13b","Type":"ContainerStarted","Data":"562161fc8fc3208d7f169a54580c32d4156bc6b12904992de9d4b1c1257a13ec"} Oct 30 22:30:04.325261 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:04.325223 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs" event={"ID":"94535d2d-703e-4804-992a-474908adf7f2","Type":"ContainerStarted","Data":"e4224be3d6fbbe6e8a602e346bad8a09265c8bc6743d18b368da3e8b8ce6e072"} Oct 30 22:30:04.450588 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:04.450531 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9" podStartSLOduration=4.450516085 podStartE2EDuration="4.450516085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:04.450422059 +0000 UTC m=+2407.002598360" watchObservedRunningTime="2026-10-30 22:30:04.450516085 +0000 UTC m=+2407.002692386" Oct 30 22:30:04.648063 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:04.648026 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9"] Oct 30 22:30:04.652495 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:04.652467 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7bb1c2-openshift-default-5dffb59bcc-tzlt9"] Oct 30 22:30:05.338482 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.338440 6006 generic.go:356] "Generic (PLEG): container finished" podID="cd8cd9b2-105d-4115-a810-44e907efaba4" containerID="f5bcf686f4eefca6ef18c8ee2f02fff92b900a85732ce0637e1a055db62af82b" exitCode=60 Oct 30 22:30:05.338968 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.338532 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg" event={"ID":"cd8cd9b2-105d-4115-a810-44e907efaba4","Type":"ContainerDied","Data":"f5bcf686f4eefca6ef18c8ee2f02fff92b900a85732ce0637e1a055db62af82b"} Oct 30 22:30:05.338968 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.338645 6006 scope.go:122] "RemoveContainer" containerID="a884347d6606136dd2a5e2d21ed4b7935ba30f96e0af08c0236829af93b77e5f" Oct 30 22:30:05.339104 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.339018 6006 scope.go:122] "RemoveContainer" containerID="f5bcf686f4eefca6ef18c8ee2f02fff92b900a85732ce0637e1a055db62af82b" Oct 30 22:30:05.339409 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:05.339382 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg_odh-ai-gateway-infra(cd8cd9b2-105d-4115-a810-44e907efaba4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg" podUID="cd8cd9b2-105d-4115-a810-44e907efaba4" Oct 30 22:30:05.342094 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.342073 6006 scope.go:122] "RemoveContainer" containerID="df287728602da41b86370f06fa955c5104ff0c6cb13ebc62ad59d73306e2ea3f" Oct 30 22:30:05.342659 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:05.342465 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9_odh-ai-gateway-infra(cf68b963-460e-4af7-8f71-d00b8e0449fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9" podUID="cf68b963-460e-4af7-8f71-d00b8e0449fc" Oct 30 22:30:05.344450 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.344418 6006 generic.go:356] "Generic (PLEG): container finished" podID="d9284e65-6486-487b-a207-a1b48b7ee19b" containerID="1cb5b41eea798c8c6a7c0d03e9ca84f9c4b686f8564d6bcf9efea1550a3d7a41" exitCode=60 Oct 30 22:30:05.344569 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.344504 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r" event={"ID":"d9284e65-6486-487b-a207-a1b48b7ee19b","Type":"ContainerDied","Data":"1cb5b41eea798c8c6a7c0d03e9ca84f9c4b686f8564d6bcf9efea1550a3d7a41"} Oct 30 22:30:05.344814 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.344797 6006 scope.go:122] "RemoveContainer" containerID="1cb5b41eea798c8c6a7c0d03e9ca84f9c4b686f8564d6bcf9efea1550a3d7a41" Oct 30 22:30:05.345159 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:05.345137 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r_odh-ai-gateway-infra(d9284e65-6486-487b-a207-a1b48b7ee19b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r" podUID="d9284e65-6486-487b-a207-a1b48b7ee19b" Oct 30 22:30:05.347593 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.347565 6006 generic.go:356] "Generic (PLEG): container finished" podID="34557f8d-a341-4f36-9914-8ec5a9af1228" containerID="653f41141456ee5c763761b2caca573eaab8245be67e5a00297a5d67e4e7a0ab" exitCode=60 Oct 30 22:30:05.347789 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.347681 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q" event={"ID":"34557f8d-a341-4f36-9914-8ec5a9af1228","Type":"ContainerDied","Data":"653f41141456ee5c763761b2caca573eaab8245be67e5a00297a5d67e4e7a0ab"} Oct 30 22:30:05.347965 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.347898 6006 scope.go:122] "RemoveContainer" containerID="653f41141456ee5c763761b2caca573eaab8245be67e5a00297a5d67e4e7a0ab" Oct 30 22:30:05.348355 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:05.348332 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q_odh-ai-gateway-infra(34557f8d-a341-4f36-9914-8ec5a9af1228)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q" podUID="34557f8d-a341-4f36-9914-8ec5a9af1228" Oct 30 22:30:05.351562 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.351536 6006 generic.go:356] "Generic (PLEG): container finished" podID="fece6ef8-ca00-4666-9860-0383ee19082c" containerID="7888768b65a36cbc525bdb182847eba55143cd4bd0666020baca01a133d8a41e" exitCode=60 Oct 30 22:30:05.351702 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.351675 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5" event={"ID":"fece6ef8-ca00-4666-9860-0383ee19082c","Type":"ContainerDied","Data":"7888768b65a36cbc525bdb182847eba55143cd4bd0666020baca01a133d8a41e"} Oct 30 22:30:05.352031 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.352010 6006 scope.go:122] "RemoveContainer" containerID="7888768b65a36cbc525bdb182847eba55143cd4bd0666020baca01a133d8a41e" Oct 30 22:30:05.352411 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:05.352385 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5_odh-ai-gateway-infra(fece6ef8-ca00-4666-9860-0383ee19082c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5" podUID="fece6ef8-ca00-4666-9860-0383ee19082c" Oct 30 22:30:05.353892 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.353868 6006 generic.go:356] "Generic (PLEG): container finished" podID="92efe8ed-fd67-4f56-a4d0-a7ef63e8964f" containerID="9c7add8c86095b54e435a13dca50fd7198a8a2642ead825a38ed925923aa4654" exitCode=60 Oct 30 22:30:05.353993 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.353952 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k" event={"ID":"92efe8ed-fd67-4f56-a4d0-a7ef63e8964f","Type":"ContainerDied","Data":"9c7add8c86095b54e435a13dca50fd7198a8a2642ead825a38ed925923aa4654"} Oct 30 22:30:05.354198 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.354181 6006 scope.go:122] "RemoveContainer" containerID="9c7add8c86095b54e435a13dca50fd7198a8a2642ead825a38ed925923aa4654" Oct 30 22:30:05.355106 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:05.355075 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k_odh-ai-gateway-infra(92efe8ed-fd67-4f56-a4d0-a7ef63e8964f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k" podUID="92efe8ed-fd67-4f56-a4d0-a7ef63e8964f" Oct 30 22:30:05.357811 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.357794 6006 scope.go:122] "RemoveContainer" containerID="fb3c0dd6f149cc322de3f89088d8411bc1ec1cda9b2954662772cedc78a208fb" Oct 30 22:30:05.359203 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.359178 6006 generic.go:356] "Generic (PLEG): container finished" podID="dcc289f1-3dd7-4182-bc07-8ca05b24b13b" containerID="562161fc8fc3208d7f169a54580c32d4156bc6b12904992de9d4b1c1257a13ec" exitCode=60 Oct 30 22:30:05.359295 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.359269 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5" event={"ID":"dcc289f1-3dd7-4182-bc07-8ca05b24b13b","Type":"ContainerDied","Data":"562161fc8fc3208d7f169a54580c32d4156bc6b12904992de9d4b1c1257a13ec"} Oct 30 22:30:05.359472 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.359458 6006 scope.go:122] "RemoveContainer" containerID="562161fc8fc3208d7f169a54580c32d4156bc6b12904992de9d4b1c1257a13ec" Oct 30 22:30:05.359676 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:05.359657 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5_odh-ai-gateway-infra(dcc289f1-3dd7-4182-bc07-8ca05b24b13b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5" podUID="dcc289f1-3dd7-4182-bc07-8ca05b24b13b" Oct 30 22:30:05.361083 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.361064 6006 generic.go:356] "Generic (PLEG): container finished" podID="08446265-b751-4ff4-a788-488684eba3ea" containerID="da4e13fc06569c49f2a6482efec222798572d4967dd5256797e588b43b2aa997" exitCode=60 Oct 30 22:30:05.361145 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.361119 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9" event={"ID":"08446265-b751-4ff4-a788-488684eba3ea","Type":"ContainerDied","Data":"da4e13fc06569c49f2a6482efec222798572d4967dd5256797e588b43b2aa997"} Oct 30 22:30:05.361364 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.361350 6006 scope.go:122] "RemoveContainer" containerID="da4e13fc06569c49f2a6482efec222798572d4967dd5256797e588b43b2aa997" Oct 30 22:30:05.361580 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:05.361558 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9_odh-ai-gateway-infra(08446265-b751-4ff4-a788-488684eba3ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9" podUID="08446265-b751-4ff4-a788-488684eba3ea" Oct 30 22:30:05.363013 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.362988 6006 generic.go:356] "Generic (PLEG): container finished" podID="94535d2d-703e-4804-992a-474908adf7f2" containerID="e4224be3d6fbbe6e8a602e346bad8a09265c8bc6743d18b368da3e8b8ce6e072" exitCode=60 Oct 30 22:30:05.363085 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.363027 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs" event={"ID":"94535d2d-703e-4804-992a-474908adf7f2","Type":"ContainerDied","Data":"e4224be3d6fbbe6e8a602e346bad8a09265c8bc6743d18b368da3e8b8ce6e072"} Oct 30 22:30:05.363233 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.363221 6006 scope.go:122] "RemoveContainer" containerID="e4224be3d6fbbe6e8a602e346bad8a09265c8bc6743d18b368da3e8b8ce6e072" Oct 30 22:30:05.363443 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:05.363423 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs_odh-ai-gateway-infra(94535d2d-703e-4804-992a-474908adf7f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs" podUID="94535d2d-703e-4804-992a-474908adf7f2" Oct 30 22:30:05.371799 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.371772 6006 scope.go:122] "RemoveContainer" containerID="e05e596000bf50155a63b74dafe935a001f34a2fc2e77b81a0f65c7447ad5d00" Oct 30 22:30:05.385583 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.385557 6006 scope.go:122] "RemoveContainer" containerID="73f6654c6cb541a2b9e2b81e2b1df42648d9a59745541546c89385ae4fb33c10" Oct 30 22:30:05.400306 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.400278 6006 scope.go:122] "RemoveContainer" containerID="87894f30b5c647fa35b16467bf326ec1c6fa17880e9970e1d4eede5e725fcdae" Oct 30 22:30:05.426573 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.426548 6006 scope.go:122] "RemoveContainer" containerID="20ea3d5fd09f9ca3345aa72d88c7c13b5fb899e38bc529c0dca73e066305e2e9" Oct 30 22:30:05.443115 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.443086 6006 scope.go:122] "RemoveContainer" containerID="7cac426c5d1a3017c40c2fcec6db79be493ad010edd65202c71beebf1b73d9e8" Oct 30 22:30:05.458729 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.458705 6006 scope.go:122] "RemoveContainer" containerID="0e312043a1734ab3c101241289e5ae6ea691dbfe4835f97c3f71c61920a14633" Oct 30 22:30:05.606852 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:05.606764 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f33ebace-e652-4f07-bbce-9c334faa3948" path="/var/lib/kubelet/pods/f33ebace-e652-4f07-bbce-9c334faa3948/volumes" Oct 30 22:30:06.373222 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:06.373178 6006 scope.go:122] "RemoveContainer" containerID="f5bcf686f4eefca6ef18c8ee2f02fff92b900a85732ce0637e1a055db62af82b" Oct 30 22:30:06.373740 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:06.373496 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg_odh-ai-gateway-infra(cd8cd9b2-105d-4115-a810-44e907efaba4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg" podUID="cd8cd9b2-105d-4115-a810-44e907efaba4" Oct 30 22:30:06.375047 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:06.375019 6006 scope.go:122] "RemoveContainer" containerID="1cb5b41eea798c8c6a7c0d03e9ca84f9c4b686f8564d6bcf9efea1550a3d7a41" Oct 30 22:30:06.375280 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:06.375242 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r_odh-ai-gateway-infra(d9284e65-6486-487b-a207-a1b48b7ee19b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r" podUID="d9284e65-6486-487b-a207-a1b48b7ee19b" Oct 30 22:30:06.376885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:06.376862 6006 scope.go:122] "RemoveContainer" containerID="653f41141456ee5c763761b2caca573eaab8245be67e5a00297a5d67e4e7a0ab" Oct 30 22:30:06.377141 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:06.377118 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q_odh-ai-gateway-infra(34557f8d-a341-4f36-9914-8ec5a9af1228)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q" podUID="34557f8d-a341-4f36-9914-8ec5a9af1228" Oct 30 22:30:06.378506 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:06.378484 6006 scope.go:122] "RemoveContainer" containerID="7888768b65a36cbc525bdb182847eba55143cd4bd0666020baca01a133d8a41e" Oct 30 22:30:06.378722 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:06.378702 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5_odh-ai-gateway-infra(fece6ef8-ca00-4666-9860-0383ee19082c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5" podUID="fece6ef8-ca00-4666-9860-0383ee19082c" Oct 30 22:30:06.380171 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:06.380146 6006 scope.go:122] "RemoveContainer" containerID="9c7add8c86095b54e435a13dca50fd7198a8a2642ead825a38ed925923aa4654" Oct 30 22:30:06.380630 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:06.380607 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k_odh-ai-gateway-infra(92efe8ed-fd67-4f56-a4d0-a7ef63e8964f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k" podUID="92efe8ed-fd67-4f56-a4d0-a7ef63e8964f" Oct 30 22:30:06.381828 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:06.381811 6006 scope.go:122] "RemoveContainer" containerID="562161fc8fc3208d7f169a54580c32d4156bc6b12904992de9d4b1c1257a13ec" Oct 30 22:30:06.382007 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:06.381989 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5_odh-ai-gateway-infra(dcc289f1-3dd7-4182-bc07-8ca05b24b13b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5" podUID="dcc289f1-3dd7-4182-bc07-8ca05b24b13b" Oct 30 22:30:06.383457 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:06.383439 6006 scope.go:122] "RemoveContainer" containerID="da4e13fc06569c49f2a6482efec222798572d4967dd5256797e588b43b2aa997" Oct 30 22:30:06.383676 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:06.383655 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9_odh-ai-gateway-infra(08446265-b751-4ff4-a788-488684eba3ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9" podUID="08446265-b751-4ff4-a788-488684eba3ea" Oct 30 22:30:06.384917 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:06.384898 6006 scope.go:122] "RemoveContainer" containerID="e4224be3d6fbbe6e8a602e346bad8a09265c8bc6743d18b368da3e8b8ce6e072" Oct 30 22:30:06.385099 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:06.385080 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs_odh-ai-gateway-infra(94535d2d-703e-4804-992a-474908adf7f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs" podUID="94535d2d-703e-4804-992a-474908adf7f2" Oct 30 22:30:11.325363 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:11.325323 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4"] Oct 30 22:30:11.325790 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:11.325553 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" podUID="88f78b1a-a774-44f1-8a64-8786dcdcf528" containerName="istio-proxy" containerID="cri-o://47eee70ff05a9a50b15660f5d6609211bd00f3e32e1344a24e7891893dc87462" gracePeriod=30 Oct 30 22:30:15.605719 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:15.605682 6006 scope.go:122] "RemoveContainer" containerID="9c7add8c86095b54e435a13dca50fd7198a8a2642ead825a38ed925923aa4654" Oct 30 22:30:15.606363 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:15.605862 6006 scope.go:122] "RemoveContainer" containerID="df287728602da41b86370f06fa955c5104ff0c6cb13ebc62ad59d73306e2ea3f" Oct 30 22:30:15.606363 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:15.605974 6006 scope.go:122] "RemoveContainer" containerID="653f41141456ee5c763761b2caca573eaab8245be67e5a00297a5d67e4e7a0ab" Oct 30 22:30:15.606363 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:15.606308 6006 scope.go:122] "RemoveContainer" containerID="f5bcf686f4eefca6ef18c8ee2f02fff92b900a85732ce0637e1a055db62af82b" Oct 30 22:30:16.465959 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.465871 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k" event={"ID":"92efe8ed-fd67-4f56-a4d0-a7ef63e8964f","Type":"ContainerStarted","Data":"8380d0ddbd90ca72b5a073897b8f38ec972d0af402b6706b5f2a1ce5b335f439"} Oct 30 22:30:16.468365 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.468329 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg" event={"ID":"cd8cd9b2-105d-4115-a810-44e907efaba4","Type":"ContainerStarted","Data":"2f5be2a24a843ceef5f6deefb607e3ef2c840cfd2827a08385daa00dada6bd0c"} Oct 30 22:30:16.470708 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.470676 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9" event={"ID":"cf68b963-460e-4af7-8f71-d00b8e0449fc","Type":"ContainerStarted","Data":"24dee1d948a7cfe7df6c7a624d575d499c51b30d7b8816051d39ae802eab9fe4"} Oct 30 22:30:16.472780 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.472753 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q" event={"ID":"34557f8d-a341-4f36-9914-8ec5a9af1228","Type":"ContainerStarted","Data":"54ff7d1582de3050230994500de0ed127e5e54334341f8df8cc009f3308814f3"} Oct 30 22:30:16.474842 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.474818 6006 generic.go:356] "Generic (PLEG): container finished" podID="88f78b1a-a774-44f1-8a64-8786dcdcf528" containerID="47eee70ff05a9a50b15660f5d6609211bd00f3e32e1344a24e7891893dc87462" exitCode=0 Oct 30 22:30:16.474969 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.474853 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" event={"ID":"88f78b1a-a774-44f1-8a64-8786dcdcf528","Type":"ContainerDied","Data":"47eee70ff05a9a50b15660f5d6609211bd00f3e32e1344a24e7891893dc87462"} Oct 30 22:30:16.595918 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.595894 6006 scope.go:122] "RemoveContainer" containerID="da4e13fc06569c49f2a6482efec222798572d4967dd5256797e588b43b2aa997" Oct 30 22:30:16.620726 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.620612 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:30:16.810327 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.805070 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-token\" (UniqueName: \"kubernetes.io/projected/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-token\") pod \"88f78b1a-a774-44f1-8a64-8786dcdcf528\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " Oct 30 22:30:16.810327 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.805137 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88f78b1a-a774-44f1-8a64-8786dcdcf528-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88f78b1a-a774-44f1-8a64-8786dcdcf528-istiod-ca-cert\") pod \"88f78b1a-a774-44f1-8a64-8786dcdcf528\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " Oct 30 22:30:16.810327 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.805866 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-envoy\") pod \"88f78b1a-a774-44f1-8a64-8786dcdcf528\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " Oct 30 22:30:16.810327 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.805887 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88f78b1a-a774-44f1-8a64-8786dcdcf528-istiod-ca-cert" pod "88f78b1a-a774-44f1-8a64-8786dcdcf528" (UID: "88f78b1a-a774-44f1-8a64-8786dcdcf528"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:16.810327 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.806306 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-podinfo\") pod \"88f78b1a-a774-44f1-8a64-8786dcdcf528\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " Oct 30 22:30:16.810327 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.806411 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-data\") pod \"88f78b1a-a774-44f1-8a64-8786dcdcf528\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " Oct 30 22:30:16.810327 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.806450 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-workload-socket\") pod \"88f78b1a-a774-44f1-8a64-8786dcdcf528\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " Oct 30 22:30:16.810327 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.806560 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-workload-certs\") pod \"88f78b1a-a774-44f1-8a64-8786dcdcf528\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " Oct 30 22:30:16.810327 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.806600 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-credential-socket\") pod \"88f78b1a-a774-44f1-8a64-8786dcdcf528\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " Oct 30 22:30:16.810327 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.806691 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88f78b1a-a774-44f1-8a64-8786dcdcf528-kube-api-access-gjhpn\" (UniqueName: \"kubernetes.io/projected/88f78b1a-a774-44f1-8a64-8786dcdcf528-kube-api-access-gjhpn\") pod \"88f78b1a-a774-44f1-8a64-8786dcdcf528\" (UID: \"88f78b1a-a774-44f1-8a64-8786dcdcf528\") " Oct 30 22:30:16.810327 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.807181 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88f78b1a-a774-44f1-8a64-8786dcdcf528-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:16.810327 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.807840 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-workload-certs" pod "88f78b1a-a774-44f1-8a64-8786dcdcf528" (UID: "88f78b1a-a774-44f1-8a64-8786dcdcf528"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:16.810327 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.808082 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-credential-socket" pod "88f78b1a-a774-44f1-8a64-8786dcdcf528" (UID: "88f78b1a-a774-44f1-8a64-8786dcdcf528"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:16.810327 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.808265 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-data" pod "88f78b1a-a774-44f1-8a64-8786dcdcf528" (UID: "88f78b1a-a774-44f1-8a64-8786dcdcf528"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:16.810327 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.808422 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-workload-socket" pod "88f78b1a-a774-44f1-8a64-8786dcdcf528" (UID: "88f78b1a-a774-44f1-8a64-8786dcdcf528"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:16.810327 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.808625 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-token" pod "88f78b1a-a774-44f1-8a64-8786dcdcf528" (UID: "88f78b1a-a774-44f1-8a64-8786dcdcf528"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:16.810327 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.809432 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-envoy" pod "88f78b1a-a774-44f1-8a64-8786dcdcf528" (UID: "88f78b1a-a774-44f1-8a64-8786dcdcf528"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:16.811394 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.810815 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f78b1a-a774-44f1-8a64-8786dcdcf528-kube-api-access-gjhpn" pod "88f78b1a-a774-44f1-8a64-8786dcdcf528" (UID: "88f78b1a-a774-44f1-8a64-8786dcdcf528"). InnerVolumeSpecName "kube-api-access-gjhpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:16.811394 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.810845 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-podinfo" pod "88f78b1a-a774-44f1-8a64-8786dcdcf528" (UID: "88f78b1a-a774-44f1-8a64-8786dcdcf528"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:30:16.908153 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.908118 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:16.908153 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.908154 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:16.908153 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.908167 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:16.908153 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.908180 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:16.908708 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.908192 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:16.908708 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.908205 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f78b1a-a774-44f1-8a64-8786dcdcf528-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:16.908708 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.908335 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjhpn\" (UniqueName: \"kubernetes.io/projected/88f78b1a-a774-44f1-8a64-8786dcdcf528-kube-api-access-gjhpn\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:16.908708 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:16.908349 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f78b1a-a774-44f1-8a64-8786dcdcf528-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:17.483787 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.483692 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9" event={"ID":"08446265-b751-4ff4-a788-488684eba3ea","Type":"ContainerStarted","Data":"11bb0667860b9fc44e6582f888e347671d94f3f6fc2d305cef802eb861cb1e0f"} Oct 30 22:30:17.485740 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.485711 6006 generic.go:356] "Generic (PLEG): container finished" podID="cd8cd9b2-105d-4115-a810-44e907efaba4" containerID="2f5be2a24a843ceef5f6deefb607e3ef2c840cfd2827a08385daa00dada6bd0c" exitCode=60 Oct 30 22:30:17.485876 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.485782 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg" event={"ID":"cd8cd9b2-105d-4115-a810-44e907efaba4","Type":"ContainerDied","Data":"2f5be2a24a843ceef5f6deefb607e3ef2c840cfd2827a08385daa00dada6bd0c"} Oct 30 22:30:17.485876 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.485837 6006 scope.go:122] "RemoveContainer" containerID="f5bcf686f4eefca6ef18c8ee2f02fff92b900a85732ce0637e1a055db62af82b" Oct 30 22:30:17.486223 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.486200 6006 scope.go:122] "RemoveContainer" containerID="2f5be2a24a843ceef5f6deefb607e3ef2c840cfd2827a08385daa00dada6bd0c" Oct 30 22:30:17.486548 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:17.486517 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg_odh-ai-gateway-infra(cd8cd9b2-105d-4115-a810-44e907efaba4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg" podUID="cd8cd9b2-105d-4115-a810-44e907efaba4" Oct 30 22:30:17.487955 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.487930 6006 generic.go:356] "Generic (PLEG): container finished" podID="cf68b963-460e-4af7-8f71-d00b8e0449fc" containerID="24dee1d948a7cfe7df6c7a624d575d499c51b30d7b8816051d39ae802eab9fe4" exitCode=60 Oct 30 22:30:17.488058 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.488007 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9" event={"ID":"cf68b963-460e-4af7-8f71-d00b8e0449fc","Type":"ContainerDied","Data":"24dee1d948a7cfe7df6c7a624d575d499c51b30d7b8816051d39ae802eab9fe4"} Oct 30 22:30:17.488508 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.488486 6006 scope.go:122] "RemoveContainer" containerID="24dee1d948a7cfe7df6c7a624d575d499c51b30d7b8816051d39ae802eab9fe4" Oct 30 22:30:17.488756 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:17.488735 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9_odh-ai-gateway-infra(cf68b963-460e-4af7-8f71-d00b8e0449fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9" podUID="cf68b963-460e-4af7-8f71-d00b8e0449fc" Oct 30 22:30:17.490067 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.490046 6006 generic.go:356] "Generic (PLEG): container finished" podID="34557f8d-a341-4f36-9914-8ec5a9af1228" containerID="54ff7d1582de3050230994500de0ed127e5e54334341f8df8cc009f3308814f3" exitCode=60 Oct 30 22:30:17.490159 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.490116 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q" event={"ID":"34557f8d-a341-4f36-9914-8ec5a9af1228","Type":"ContainerDied","Data":"54ff7d1582de3050230994500de0ed127e5e54334341f8df8cc009f3308814f3"} Oct 30 22:30:17.490444 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.490427 6006 scope.go:122] "RemoveContainer" containerID="54ff7d1582de3050230994500de0ed127e5e54334341f8df8cc009f3308814f3" Oct 30 22:30:17.490671 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:17.490651 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q_odh-ai-gateway-infra(34557f8d-a341-4f36-9914-8ec5a9af1228)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q" podUID="34557f8d-a341-4f36-9914-8ec5a9af1228" Oct 30 22:30:17.491677 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.491654 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" Oct 30 22:30:17.491677 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.491670 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4" event={"ID":"88f78b1a-a774-44f1-8a64-8786dcdcf528","Type":"ContainerDied","Data":"64053369c6fa22cdfe07d4bf392b5a7a480fd158364284440c70155d67203248"} Oct 30 22:30:17.493751 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.493731 6006 generic.go:356] "Generic (PLEG): container finished" podID="92efe8ed-fd67-4f56-a4d0-a7ef63e8964f" containerID="8380d0ddbd90ca72b5a073897b8f38ec972d0af402b6706b5f2a1ce5b335f439" exitCode=60 Oct 30 22:30:17.493851 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.493774 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k" event={"ID":"92efe8ed-fd67-4f56-a4d0-a7ef63e8964f","Type":"ContainerDied","Data":"8380d0ddbd90ca72b5a073897b8f38ec972d0af402b6706b5f2a1ce5b335f439"} Oct 30 22:30:17.494091 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.494076 6006 scope.go:122] "RemoveContainer" containerID="8380d0ddbd90ca72b5a073897b8f38ec972d0af402b6706b5f2a1ce5b335f439" Oct 30 22:30:17.494344 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:17.494316 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k_odh-ai-gateway-infra(92efe8ed-fd67-4f56-a4d0-a7ef63e8964f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k" podUID="92efe8ed-fd67-4f56-a4d0-a7ef63e8964f" Oct 30 22:30:17.502805 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.502783 6006 scope.go:122] "RemoveContainer" containerID="df287728602da41b86370f06fa955c5104ff0c6cb13ebc62ad59d73306e2ea3f" Oct 30 22:30:17.516902 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.516876 6006 scope.go:122] "RemoveContainer" containerID="653f41141456ee5c763761b2caca573eaab8245be67e5a00297a5d67e4e7a0ab" Oct 30 22:30:17.531697 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.531676 6006 scope.go:122] "RemoveContainer" containerID="47eee70ff05a9a50b15660f5d6609211bd00f3e32e1344a24e7891893dc87462" Oct 30 22:30:17.547432 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.547394 6006 scope.go:122] "RemoveContainer" containerID="9c7add8c86095b54e435a13dca50fd7198a8a2642ead825a38ed925923aa4654" Oct 30 22:30:17.595664 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.595632 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k"] Oct 30 22:30:17.602352 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.602320 6006 scope.go:122] "RemoveContainer" containerID="1cb5b41eea798c8c6a7c0d03e9ca84f9c4b686f8564d6bcf9efea1550a3d7a41" Oct 30 22:30:17.602734 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.602569 6006 scope.go:122] "RemoveContainer" containerID="e4224be3d6fbbe6e8a602e346bad8a09265c8bc6743d18b368da3e8b8ce6e072" Oct 30 22:30:17.704870 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.704824 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg"] Oct 30 22:30:17.811693 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.811659 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9"] Oct 30 22:30:17.923422 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.923387 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q"] Oct 30 22:30:17.996792 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:17.996742 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4"] Oct 30 22:30:18.014269 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:18.014208 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a00d9b-openshift-default-847df8fc59-86wk4"] Oct 30 22:30:18.512064 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:18.512028 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r" event={"ID":"d9284e65-6486-487b-a207-a1b48b7ee19b","Type":"ContainerStarted","Data":"8f3ebd13e04ccc9bb4333528c5d325aef7a02e7a125f4a46e10cdd2ffdd94f5e"} Oct 30 22:30:18.524278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:18.523703 6006 generic.go:356] "Generic (PLEG): container finished" podID="08446265-b751-4ff4-a788-488684eba3ea" containerID="11bb0667860b9fc44e6582f888e347671d94f3f6fc2d305cef802eb861cb1e0f" exitCode=60 Oct 30 22:30:18.524278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:18.523794 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9" event={"ID":"08446265-b751-4ff4-a788-488684eba3ea","Type":"ContainerDied","Data":"11bb0667860b9fc44e6582f888e347671d94f3f6fc2d305cef802eb861cb1e0f"} Oct 30 22:30:18.524278 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:18.523846 6006 scope.go:122] "RemoveContainer" containerID="da4e13fc06569c49f2a6482efec222798572d4967dd5256797e588b43b2aa997" Oct 30 22:30:18.525491 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:18.524714 6006 scope.go:122] "RemoveContainer" containerID="11bb0667860b9fc44e6582f888e347671d94f3f6fc2d305cef802eb861cb1e0f" Oct 30 22:30:18.526203 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:18.525802 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9_odh-ai-gateway-infra(08446265-b751-4ff4-a788-488684eba3ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9" podUID="08446265-b751-4ff4-a788-488684eba3ea" Oct 30 22:30:18.531279 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:18.531209 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs" event={"ID":"94535d2d-703e-4804-992a-474908adf7f2","Type":"ContainerStarted","Data":"52a1908fe43e0049f13dd6079467b356ca084fd6b491e5527fbbdc72b9ce0025"} Oct 30 22:30:18.608758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:18.608725 6006 scope.go:122] "RemoveContainer" containerID="562161fc8fc3208d7f169a54580c32d4156bc6b12904992de9d4b1c1257a13ec" Oct 30 22:30:18.647132 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:18.647090 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9"] Oct 30 22:30:19.051450 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.051416 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg" Oct 30 22:30:19.055967 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.055936 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k" Oct 30 22:30:19.060777 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.060752 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9" Oct 30 22:30:19.066083 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.066061 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q" Oct 30 22:30:19.205299 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.205239 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd8cd9b2-105d-4115-a810-44e907efaba4-kube-api-access-slpbr\" (UniqueName: \"kubernetes.io/projected/cd8cd9b2-105d-4115-a810-44e907efaba4-kube-api-access-slpbr\") pod \"cd8cd9b2-105d-4115-a810-44e907efaba4\" (UID: \"cd8cd9b2-105d-4115-a810-44e907efaba4\") " Oct 30 22:30:19.205461 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.205435 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34557f8d-a341-4f36-9914-8ec5a9af1228-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34557f8d-a341-4f36-9914-8ec5a9af1228-maas-api-service-ca\") pod \"34557f8d-a341-4f36-9914-8ec5a9af1228\" (UID: \"34557f8d-a341-4f36-9914-8ec5a9af1228\") " Oct 30 22:30:19.205514 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.205504 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf68b963-460e-4af7-8f71-d00b8e0449fc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf68b963-460e-4af7-8f71-d00b8e0449fc-maas-api-service-ca\") pod \"cf68b963-460e-4af7-8f71-d00b8e0449fc\" (UID: \"cf68b963-460e-4af7-8f71-d00b8e0449fc\") " Oct 30 22:30:19.205946 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.205609 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf68b963-460e-4af7-8f71-d00b8e0449fc-kube-api-access-b8mcm\" (UniqueName: \"kubernetes.io/projected/cf68b963-460e-4af7-8f71-d00b8e0449fc-kube-api-access-b8mcm\") pod \"cf68b963-460e-4af7-8f71-d00b8e0449fc\" (UID: \"cf68b963-460e-4af7-8f71-d00b8e0449fc\") " Oct 30 22:30:19.205946 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.205806 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd8cd9b2-105d-4115-a810-44e907efaba4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8cd9b2-105d-4115-a810-44e907efaba4-maas-api-service-ca\") pod \"cd8cd9b2-105d-4115-a810-44e907efaba4\" (UID: \"cd8cd9b2-105d-4115-a810-44e907efaba4\") " Oct 30 22:30:19.205946 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.205847 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34557f8d-a341-4f36-9914-8ec5a9af1228-maas-api-service-ca" pod "34557f8d-a341-4f36-9914-8ec5a9af1228" (UID: "34557f8d-a341-4f36-9914-8ec5a9af1228"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:19.205946 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.205859 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf68b963-460e-4af7-8f71-d00b8e0449fc-maas-api-service-ca" pod "cf68b963-460e-4af7-8f71-d00b8e0449fc" (UID: "cf68b963-460e-4af7-8f71-d00b8e0449fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:19.205946 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.205864 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34557f8d-a341-4f36-9914-8ec5a9af1228-kube-api-access-m5vbz\" (UniqueName: \"kubernetes.io/projected/34557f8d-a341-4f36-9914-8ec5a9af1228-kube-api-access-m5vbz\") pod \"34557f8d-a341-4f36-9914-8ec5a9af1228\" (UID: \"34557f8d-a341-4f36-9914-8ec5a9af1228\") " Oct 30 22:30:19.205946 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.205932 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92efe8ed-fd67-4f56-a4d0-a7ef63e8964f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92efe8ed-fd67-4f56-a4d0-a7ef63e8964f-maas-api-service-ca\") pod \"92efe8ed-fd67-4f56-a4d0-a7ef63e8964f\" (UID: \"92efe8ed-fd67-4f56-a4d0-a7ef63e8964f\") " Oct 30 22:30:19.206399 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.205991 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92efe8ed-fd67-4f56-a4d0-a7ef63e8964f-kube-api-access-bc92x\" (UniqueName: \"kubernetes.io/projected/92efe8ed-fd67-4f56-a4d0-a7ef63e8964f-kube-api-access-bc92x\") pod \"92efe8ed-fd67-4f56-a4d0-a7ef63e8964f\" (UID: \"92efe8ed-fd67-4f56-a4d0-a7ef63e8964f\") " Oct 30 22:30:19.206458 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.206409 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92efe8ed-fd67-4f56-a4d0-a7ef63e8964f-maas-api-service-ca" pod "92efe8ed-fd67-4f56-a4d0-a7ef63e8964f" (UID: "92efe8ed-fd67-4f56-a4d0-a7ef63e8964f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:19.206708 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.206684 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92efe8ed-fd67-4f56-a4d0-a7ef63e8964f-maas-api-service-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.206819 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.206713 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34557f8d-a341-4f36-9914-8ec5a9af1228-maas-api-service-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.206819 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.206729 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf68b963-460e-4af7-8f71-d00b8e0449fc-maas-api-service-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.207919 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.207890 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd8cd9b2-105d-4115-a810-44e907efaba4-kube-api-access-slpbr" pod "cd8cd9b2-105d-4115-a810-44e907efaba4" (UID: "cd8cd9b2-105d-4115-a810-44e907efaba4"). InnerVolumeSpecName "kube-api-access-slpbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:19.208000 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.207949 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd8cd9b2-105d-4115-a810-44e907efaba4-maas-api-service-ca" pod "cd8cd9b2-105d-4115-a810-44e907efaba4" (UID: "cd8cd9b2-105d-4115-a810-44e907efaba4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:19.208581 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.208556 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf68b963-460e-4af7-8f71-d00b8e0449fc-kube-api-access-b8mcm" pod "cf68b963-460e-4af7-8f71-d00b8e0449fc" (UID: "cf68b963-460e-4af7-8f71-d00b8e0449fc"). InnerVolumeSpecName "kube-api-access-b8mcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:19.208656 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.208587 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34557f8d-a341-4f36-9914-8ec5a9af1228-kube-api-access-m5vbz" pod "34557f8d-a341-4f36-9914-8ec5a9af1228" (UID: "34557f8d-a341-4f36-9914-8ec5a9af1228"). InnerVolumeSpecName "kube-api-access-m5vbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:19.208722 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.208702 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92efe8ed-fd67-4f56-a4d0-a7ef63e8964f-kube-api-access-bc92x" pod "92efe8ed-fd67-4f56-a4d0-a7ef63e8964f" (UID: "92efe8ed-fd67-4f56-a4d0-a7ef63e8964f"). InnerVolumeSpecName "kube-api-access-bc92x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:19.308270 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.308222 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8mcm\" (UniqueName: \"kubernetes.io/projected/cf68b963-460e-4af7-8f71-d00b8e0449fc-kube-api-access-b8mcm\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.308456 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.308299 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8cd9b2-105d-4115-a810-44e907efaba4-maas-api-service-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.308456 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.308315 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5vbz\" (UniqueName: \"kubernetes.io/projected/34557f8d-a341-4f36-9914-8ec5a9af1228-kube-api-access-m5vbz\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.308456 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.308329 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc92x\" (UniqueName: \"kubernetes.io/projected/92efe8ed-fd67-4f56-a4d0-a7ef63e8964f-kube-api-access-bc92x\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.308456 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.308341 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slpbr\" (UniqueName: \"kubernetes.io/projected/cd8cd9b2-105d-4115-a810-44e907efaba4-kube-api-access-slpbr\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.541129 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.541101 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9" Oct 30 22:30:19.541362 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.541137 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9" event={"ID":"cf68b963-460e-4af7-8f71-d00b8e0449fc","Type":"ContainerDied","Data":"f0f09e0b35600e0620ffcfec9d6698d732d6b0936690217c3be924e96d492236"} Oct 30 22:30:19.541362 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.541222 6006 scope.go:122] "RemoveContainer" containerID="24dee1d948a7cfe7df6c7a624d575d499c51b30d7b8816051d39ae802eab9fe4" Oct 30 22:30:19.550481 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.550451 6006 generic.go:356] "Generic (PLEG): container finished" podID="d9284e65-6486-487b-a207-a1b48b7ee19b" containerID="8f3ebd13e04ccc9bb4333528c5d325aef7a02e7a125f4a46e10cdd2ffdd94f5e" exitCode=60 Oct 30 22:30:19.550617 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.550545 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r" event={"ID":"d9284e65-6486-487b-a207-a1b48b7ee19b","Type":"ContainerDied","Data":"8f3ebd13e04ccc9bb4333528c5d325aef7a02e7a125f4a46e10cdd2ffdd94f5e"} Oct 30 22:30:19.551365 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.551331 6006 scope.go:122] "RemoveContainer" containerID="8f3ebd13e04ccc9bb4333528c5d325aef7a02e7a125f4a46e10cdd2ffdd94f5e" Oct 30 22:30:19.551824 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:19.551794 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r_odh-ai-gateway-infra(d9284e65-6486-487b-a207-a1b48b7ee19b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r" podUID="d9284e65-6486-487b-a207-a1b48b7ee19b" Oct 30 22:30:19.552659 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.552641 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q" Oct 30 22:30:19.552781 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.552684 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q" event={"ID":"34557f8d-a341-4f36-9914-8ec5a9af1228","Type":"ContainerDied","Data":"1e7699c03393d4aed87660d523248b2cd105383ee41171f857b6fc9121f14490"} Oct 30 22:30:19.554304 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.554283 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k" event={"ID":"92efe8ed-fd67-4f56-a4d0-a7ef63e8964f","Type":"ContainerDied","Data":"edbcd2e167fa4ee3da44a446ccb3171b7e047d5871d43a2ad570beb271f51c3e"} Oct 30 22:30:19.554392 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.554301 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k" Oct 30 22:30:19.564878 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.564841 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5" event={"ID":"dcc289f1-3dd7-4182-bc07-8ca05b24b13b","Type":"ContainerStarted","Data":"53280af433f6599b1175e56c6876871e73cd2fe5cb3c49a4ed333a1afbb05c11"} Oct 30 22:30:19.567758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.567712 6006 scope.go:122] "RemoveContainer" containerID="1cb5b41eea798c8c6a7c0d03e9ca84f9c4b686f8564d6bcf9efea1550a3d7a41" Oct 30 22:30:19.574206 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.574182 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs" event={"ID":"94535d2d-703e-4804-992a-474908adf7f2","Type":"ContainerDied","Data":"52a1908fe43e0049f13dd6079467b356ca084fd6b491e5527fbbdc72b9ce0025"} Oct 30 22:30:19.574699 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.574677 6006 scope.go:122] "RemoveContainer" containerID="52a1908fe43e0049f13dd6079467b356ca084fd6b491e5527fbbdc72b9ce0025" Oct 30 22:30:19.575202 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:19.575171 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs_odh-ai-gateway-infra(94535d2d-703e-4804-992a-474908adf7f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs" podUID="94535d2d-703e-4804-992a-474908adf7f2" Oct 30 22:30:19.575340 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.575243 6006 generic.go:356] "Generic (PLEG): container finished" podID="94535d2d-703e-4804-992a-474908adf7f2" containerID="52a1908fe43e0049f13dd6079467b356ca084fd6b491e5527fbbdc72b9ce0025" exitCode=60 Oct 30 22:30:19.579876 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.579851 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg" event={"ID":"cd8cd9b2-105d-4115-a810-44e907efaba4","Type":"ContainerDied","Data":"ced9b291eda7c1b2c8a76543f9578df6e3715edaac6d42ae825d16f5fa58ef8b"} Oct 30 22:30:19.580051 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.579882 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg" Oct 30 22:30:19.588555 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.588531 6006 scope.go:122] "RemoveContainer" containerID="54ff7d1582de3050230994500de0ed127e5e54334341f8df8cc009f3308814f3" Oct 30 22:30:19.600715 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.600689 6006 scope.go:122] "RemoveContainer" containerID="8380d0ddbd90ca72b5a073897b8f38ec972d0af402b6706b5f2a1ce5b335f439" Oct 30 22:30:19.620652 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.620580 6006 scope.go:122] "RemoveContainer" containerID="e4224be3d6fbbe6e8a602e346bad8a09265c8bc6743d18b368da3e8b8ce6e072" Oct 30 22:30:19.641578 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.640679 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88f78b1a-a774-44f1-8a64-8786dcdcf528" path="/var/lib/kubelet/pods/88f78b1a-a774-44f1-8a64-8786dcdcf528/volumes" Oct 30 22:30:19.643540 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.642310 6006 scope.go:122] "RemoveContainer" containerID="7888768b65a36cbc525bdb182847eba55143cd4bd0666020baca01a133d8a41e" Oct 30 22:30:19.646039 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.645876 6006 scope.go:122] "RemoveContainer" containerID="2f5be2a24a843ceef5f6deefb607e3ef2c840cfd2827a08385daa00dada6bd0c" Oct 30 22:30:19.746242 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.746210 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs"] Oct 30 22:30:19.816370 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.814299 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9" Oct 30 22:30:19.918880 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.918837 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9"] Oct 30 22:30:19.922968 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.922407 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08446265-b751-4ff4-a788-488684eba3ea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08446265-b751-4ff4-a788-488684eba3ea-maas-api-service-ca\") pod \"08446265-b751-4ff4-a788-488684eba3ea\" (UID: \"08446265-b751-4ff4-a788-488684eba3ea\") " Oct 30 22:30:19.922968 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.922619 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08446265-b751-4ff4-a788-488684eba3ea-kube-api-access-km2c2\" (UniqueName: \"kubernetes.io/projected/08446265-b751-4ff4-a788-488684eba3ea-kube-api-access-km2c2\") pod \"08446265-b751-4ff4-a788-488684eba3ea\" (UID: \"08446265-b751-4ff4-a788-488684eba3ea\") " Oct 30 22:30:19.923265 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.923225 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08446265-b751-4ff4-a788-488684eba3ea-maas-api-service-ca" pod "08446265-b751-4ff4-a788-488684eba3ea" (UID: "08446265-b751-4ff4-a788-488684eba3ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:19.923535 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.923516 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08446265-b751-4ff4-a788-488684eba3ea-maas-api-service-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.925701 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.925669 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08446265-b751-4ff4-a788-488684eba3ea-kube-api-access-km2c2" pod "08446265-b751-4ff4-a788-488684eba3ea" (UID: "08446265-b751-4ff4-a788-488684eba3ea"). InnerVolumeSpecName "kube-api-access-km2c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:19.925896 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.925872 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9f3ea1-29889990-4p6r9"] Oct 30 22:30:19.966611 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.966574 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6"] Oct 30 22:30:19.969425 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:19.969392 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9f3ea1-nkbw6"] Oct 30 22:30:20.024685 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:20.024644 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km2c2\" (UniqueName: \"kubernetes.io/projected/08446265-b751-4ff4-a788-488684eba3ea-kube-api-access-km2c2\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:20.140571 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:20.140534 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r"] Oct 30 22:30:20.272108 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:20.272061 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q"] Oct 30 22:30:20.281460 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:20.281425 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-5686cb-29889990-xht9q"] Oct 30 22:30:20.591421 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:20.591384 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5" event={"ID":"fece6ef8-ca00-4666-9860-0383ee19082c","Type":"ContainerStarted","Data":"2679879452dca1969f61080c4eab778df0aac23ade045eab9280e140e1d17384"} Oct 30 22:30:20.594366 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:20.594337 6006 generic.go:356] "Generic (PLEG): container finished" podID="dcc289f1-3dd7-4182-bc07-8ca05b24b13b" containerID="53280af433f6599b1175e56c6876871e73cd2fe5cb3c49a4ed333a1afbb05c11" exitCode=60 Oct 30 22:30:20.594500 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:20.594407 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5" event={"ID":"dcc289f1-3dd7-4182-bc07-8ca05b24b13b","Type":"ContainerDied","Data":"53280af433f6599b1175e56c6876871e73cd2fe5cb3c49a4ed333a1afbb05c11"} Oct 30 22:30:20.594500 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:20.594464 6006 scope.go:122] "RemoveContainer" containerID="562161fc8fc3208d7f169a54580c32d4156bc6b12904992de9d4b1c1257a13ec" Oct 30 22:30:20.594812 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:20.594793 6006 scope.go:122] "RemoveContainer" containerID="53280af433f6599b1175e56c6876871e73cd2fe5cb3c49a4ed333a1afbb05c11" Oct 30 22:30:20.595140 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:20.595116 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5_odh-ai-gateway-infra(dcc289f1-3dd7-4182-bc07-8ca05b24b13b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5" podUID="dcc289f1-3dd7-4182-bc07-8ca05b24b13b" Oct 30 22:30:20.595817 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:20.595796 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9" event={"ID":"08446265-b751-4ff4-a788-488684eba3ea","Type":"ContainerDied","Data":"8f3c196011167a2be497fa53245208f0ac37ada7a181a49c5b43b81ae7fd6247"} Oct 30 22:30:20.595924 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:20.595799 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9" Oct 30 22:30:20.620821 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:20.605240 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k"] Oct 30 22:30:20.620821 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:20.612273 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c9dbe6-29889990-td46k"] Oct 30 22:30:20.620821 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:20.618632 6006 scope.go:122] "RemoveContainer" containerID="11bb0667860b9fc44e6582f888e347671d94f3f6fc2d305cef802eb861cb1e0f" Oct 30 22:30:20.722050 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:20.722012 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg"] Oct 30 22:30:20.727330 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:20.727297 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7f76c4-29889990-5clcg"] Oct 30 22:30:20.796555 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:20.796527 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs" Oct 30 22:30:20.858004 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:20.857938 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r" Oct 30 22:30:20.938268 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:20.936961 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94535d2d-703e-4804-992a-474908adf7f2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94535d2d-703e-4804-992a-474908adf7f2-maas-api-service-ca\") pod \"94535d2d-703e-4804-992a-474908adf7f2\" (UID: \"94535d2d-703e-4804-992a-474908adf7f2\") " Oct 30 22:30:20.938268 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:20.937154 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94535d2d-703e-4804-992a-474908adf7f2-kube-api-access-vlgnn\" (UniqueName: \"kubernetes.io/projected/94535d2d-703e-4804-992a-474908adf7f2-kube-api-access-vlgnn\") pod \"94535d2d-703e-4804-992a-474908adf7f2\" (UID: \"94535d2d-703e-4804-992a-474908adf7f2\") " Oct 30 22:30:20.938268 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:20.937638 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94535d2d-703e-4804-992a-474908adf7f2-maas-api-service-ca" pod "94535d2d-703e-4804-992a-474908adf7f2" (UID: "94535d2d-703e-4804-992a-474908adf7f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:20.938268 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:20.937776 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94535d2d-703e-4804-992a-474908adf7f2-maas-api-service-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:20.940042 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:20.940009 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94535d2d-703e-4804-992a-474908adf7f2-kube-api-access-vlgnn" pod "94535d2d-703e-4804-992a-474908adf7f2" (UID: "94535d2d-703e-4804-992a-474908adf7f2"). InnerVolumeSpecName "kube-api-access-vlgnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:21.000153 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.000108 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9"] Oct 30 22:30:21.013674 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.013636 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6b6a5-29889990-2pps9"] Oct 30 22:30:21.038978 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.038935 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9284e65-6486-487b-a207-a1b48b7ee19b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9284e65-6486-487b-a207-a1b48b7ee19b-maas-api-service-ca\") pod \"d9284e65-6486-487b-a207-a1b48b7ee19b\" (UID: \"d9284e65-6486-487b-a207-a1b48b7ee19b\") " Oct 30 22:30:21.039155 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.039009 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9284e65-6486-487b-a207-a1b48b7ee19b-kube-api-access-7p2sl\" (UniqueName: \"kubernetes.io/projected/d9284e65-6486-487b-a207-a1b48b7ee19b-kube-api-access-7p2sl\") pod \"d9284e65-6486-487b-a207-a1b48b7ee19b\" (UID: \"d9284e65-6486-487b-a207-a1b48b7ee19b\") " Oct 30 22:30:21.039355 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.039327 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9284e65-6486-487b-a207-a1b48b7ee19b-maas-api-service-ca" pod "d9284e65-6486-487b-a207-a1b48b7ee19b" (UID: "d9284e65-6486-487b-a207-a1b48b7ee19b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:21.039603 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.039582 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlgnn\" (UniqueName: \"kubernetes.io/projected/94535d2d-703e-4804-992a-474908adf7f2-kube-api-access-vlgnn\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:21.039700 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.039607 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9284e65-6486-487b-a207-a1b48b7ee19b-maas-api-service-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:21.041357 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.041326 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9284e65-6486-487b-a207-a1b48b7ee19b-kube-api-access-7p2sl" pod "d9284e65-6486-487b-a207-a1b48b7ee19b" (UID: "d9284e65-6486-487b-a207-a1b48b7ee19b"). InnerVolumeSpecName "kube-api-access-7p2sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:21.140698 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.140613 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p2sl\" (UniqueName: \"kubernetes.io/projected/d9284e65-6486-487b-a207-a1b48b7ee19b-kube-api-access-7p2sl\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:21.420020 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.419896 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5"] Oct 30 22:30:21.607213 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.607172 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08446265-b751-4ff4-a788-488684eba3ea" path="/var/lib/kubelet/pods/08446265-b751-4ff4-a788-488684eba3ea/volumes" Oct 30 22:30:21.607720 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.607704 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34557f8d-a341-4f36-9914-8ec5a9af1228" path="/var/lib/kubelet/pods/34557f8d-a341-4f36-9914-8ec5a9af1228/volumes" Oct 30 22:30:21.608153 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.608131 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92efe8ed-fd67-4f56-a4d0-a7ef63e8964f" path="/var/lib/kubelet/pods/92efe8ed-fd67-4f56-a4d0-a7ef63e8964f/volumes" Oct 30 22:30:21.608972 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.608950 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd8cd9b2-105d-4115-a810-44e907efaba4" path="/var/lib/kubelet/pods/cd8cd9b2-105d-4115-a810-44e907efaba4/volumes" Oct 30 22:30:21.609435 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.609417 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf68b963-460e-4af7-8f71-d00b8e0449fc" path="/var/lib/kubelet/pods/cf68b963-460e-4af7-8f71-d00b8e0449fc/volumes" Oct 30 22:30:21.609878 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.609860 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8dacf80-a617-41ac-ba6f-8c84fe6b6e90" path="/var/lib/kubelet/pods/f8dacf80-a617-41ac-ba6f-8c84fe6b6e90/volumes" Oct 30 22:30:21.629915 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.629887 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r" Oct 30 22:30:21.630074 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.629885 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r" event={"ID":"d9284e65-6486-487b-a207-a1b48b7ee19b","Type":"ContainerDied","Data":"4852be01ac0bcdbd841cd8cd6c8daac5b09b3d00c0895bac35b86967c70990fe"} Oct 30 22:30:21.630074 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.630046 6006 scope.go:122] "RemoveContainer" containerID="8f3ebd13e04ccc9bb4333528c5d325aef7a02e7a125f4a46e10cdd2ffdd94f5e" Oct 30 22:30:21.633090 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.633059 6006 generic.go:356] "Generic (PLEG): container finished" podID="fece6ef8-ca00-4666-9860-0383ee19082c" containerID="2679879452dca1969f61080c4eab778df0aac23ade045eab9280e140e1d17384" exitCode=60 Oct 30 22:30:21.633229 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.633112 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5" event={"ID":"fece6ef8-ca00-4666-9860-0383ee19082c","Type":"ContainerDied","Data":"2679879452dca1969f61080c4eab778df0aac23ade045eab9280e140e1d17384"} Oct 30 22:30:21.633713 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.633691 6006 scope.go:122] "RemoveContainer" containerID="2679879452dca1969f61080c4eab778df0aac23ade045eab9280e140e1d17384" Oct 30 22:30:21.634075 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:21.634048 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5_odh-ai-gateway-infra(fece6ef8-ca00-4666-9860-0383ee19082c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5" podUID="fece6ef8-ca00-4666-9860-0383ee19082c" Oct 30 22:30:21.638343 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.638313 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs" event={"ID":"94535d2d-703e-4804-992a-474908adf7f2","Type":"ContainerDied","Data":"6b408e2f1a5a6cbdb46ab5ffd87c316486e6ba80e3981fdb7295dfef1443ce6c"} Oct 30 22:30:21.638506 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.638352 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs" Oct 30 22:30:21.643147 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.643127 6006 scope.go:122] "RemoveContainer" containerID="7888768b65a36cbc525bdb182847eba55143cd4bd0666020baca01a133d8a41e" Oct 30 22:30:21.663576 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.663548 6006 scope.go:122] "RemoveContainer" containerID="52a1908fe43e0049f13dd6079467b356ca084fd6b491e5527fbbdc72b9ce0025" Oct 30 22:30:21.837621 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.837587 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5" Oct 30 22:30:21.876959 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.876913 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs"] Oct 30 22:30:21.883047 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.883009 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ed44da-29889990-ph5zs"] Oct 30 22:30:21.951321 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.951213 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcc289f1-3dd7-4182-bc07-8ca05b24b13b-kube-api-access-mln7n\" (UniqueName: \"kubernetes.io/projected/dcc289f1-3dd7-4182-bc07-8ca05b24b13b-kube-api-access-mln7n\") pod \"dcc289f1-3dd7-4182-bc07-8ca05b24b13b\" (UID: \"dcc289f1-3dd7-4182-bc07-8ca05b24b13b\") " Oct 30 22:30:21.951643 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.951608 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcc289f1-3dd7-4182-bc07-8ca05b24b13b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcc289f1-3dd7-4182-bc07-8ca05b24b13b-maas-api-service-ca\") pod \"dcc289f1-3dd7-4182-bc07-8ca05b24b13b\" (UID: \"dcc289f1-3dd7-4182-bc07-8ca05b24b13b\") " Oct 30 22:30:21.952365 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.952340 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcc289f1-3dd7-4182-bc07-8ca05b24b13b-maas-api-service-ca" pod "dcc289f1-3dd7-4182-bc07-8ca05b24b13b" (UID: "dcc289f1-3dd7-4182-bc07-8ca05b24b13b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:21.953885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:21.953859 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc289f1-3dd7-4182-bc07-8ca05b24b13b-kube-api-access-mln7n" pod "dcc289f1-3dd7-4182-bc07-8ca05b24b13b" (UID: "dcc289f1-3dd7-4182-bc07-8ca05b24b13b"). InnerVolumeSpecName "kube-api-access-mln7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:22.052946 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:22.052909 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mln7n\" (UniqueName: \"kubernetes.io/projected/dcc289f1-3dd7-4182-bc07-8ca05b24b13b-kube-api-access-mln7n\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:22.052946 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:22.052945 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcc289f1-3dd7-4182-bc07-8ca05b24b13b-maas-api-service-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:22.111669 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:22.111634 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r"] Oct 30 22:30:22.115757 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:22.115723 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-65e59f-29889990-hnt9r"] Oct 30 22:30:22.430634 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:22.430598 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5"] Oct 30 22:30:22.649629 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:22.649598 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5" Oct 30 22:30:22.649763 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:22.649632 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5" event={"ID":"dcc289f1-3dd7-4182-bc07-8ca05b24b13b","Type":"ContainerDied","Data":"858fa0a942aefc25aaecf608c056588d86a70d5267671591ef3a2edd96f87515"} Oct 30 22:30:22.649763 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:22.649697 6006 scope.go:122] "RemoveContainer" containerID="53280af433f6599b1175e56c6876871e73cd2fe5cb3c49a4ed333a1afbb05c11" Oct 30 22:30:22.829799 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:22.829768 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5" Oct 30 22:30:22.863758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:22.863714 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fece6ef8-ca00-4666-9860-0383ee19082c-kube-api-access-d4vvk\" (UniqueName: \"kubernetes.io/projected/fece6ef8-ca00-4666-9860-0383ee19082c-kube-api-access-d4vvk\") pod \"fece6ef8-ca00-4666-9860-0383ee19082c\" (UID: \"fece6ef8-ca00-4666-9860-0383ee19082c\") " Oct 30 22:30:22.863964 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:22.863790 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fece6ef8-ca00-4666-9860-0383ee19082c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fece6ef8-ca00-4666-9860-0383ee19082c-maas-api-service-ca\") pod \"fece6ef8-ca00-4666-9860-0383ee19082c\" (UID: \"fece6ef8-ca00-4666-9860-0383ee19082c\") " Oct 30 22:30:22.864174 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:22.864149 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fece6ef8-ca00-4666-9860-0383ee19082c-maas-api-service-ca" pod "fece6ef8-ca00-4666-9860-0383ee19082c" (UID: "fece6ef8-ca00-4666-9860-0383ee19082c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:22.864353 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:22.864339 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fece6ef8-ca00-4666-9860-0383ee19082c-maas-api-service-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:22.866043 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:22.866020 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fece6ef8-ca00-4666-9860-0383ee19082c-kube-api-access-d4vvk" pod "fece6ef8-ca00-4666-9860-0383ee19082c" (UID: "fece6ef8-ca00-4666-9860-0383ee19082c"). InnerVolumeSpecName "kube-api-access-d4vvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:22.894366 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:22.894337 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5"] Oct 30 22:30:22.903801 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:22.903768 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a00d9b-29889990-7vwp5"] Oct 30 22:30:22.965971 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:22.965874 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4vvk\" (UniqueName: \"kubernetes.io/projected/fece6ef8-ca00-4666-9860-0383ee19082c-kube-api-access-d4vvk\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:23.604487 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:23.604450 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94535d2d-703e-4804-992a-474908adf7f2" path="/var/lib/kubelet/pods/94535d2d-703e-4804-992a-474908adf7f2/volumes" Oct 30 22:30:23.604978 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:23.604962 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9284e65-6486-487b-a207-a1b48b7ee19b" path="/var/lib/kubelet/pods/d9284e65-6486-487b-a207-a1b48b7ee19b/volumes" Oct 30 22:30:23.605401 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:23.605387 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcc289f1-3dd7-4182-bc07-8ca05b24b13b" path="/var/lib/kubelet/pods/dcc289f1-3dd7-4182-bc07-8ca05b24b13b/volumes" Oct 30 22:30:23.660216 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:23.660185 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5" Oct 30 22:30:23.660453 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:23.660219 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5" event={"ID":"fece6ef8-ca00-4666-9860-0383ee19082c","Type":"ContainerDied","Data":"f01dbd394e478dea84d369ed6b5a01b24b58c853b18d234bd3d946c4fac0b9a5"} Oct 30 22:30:23.660453 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:23.660314 6006 scope.go:122] "RemoveContainer" containerID="2679879452dca1969f61080c4eab778df0aac23ade045eab9280e140e1d17384" Oct 30 22:30:23.818937 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:23.818901 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5"] Oct 30 22:30:23.825667 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:23.825629 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-943d8f-29889990-xf9w5"] Oct 30 22:30:25.604432 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:25.604392 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fece6ef8-ca00-4666-9860-0383ee19082c" path="/var/lib/kubelet/pods/fece6ef8-ca00-4666-9860-0383ee19082c/volumes" Oct 30 22:30:26.348465 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:26.348423 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6"] Oct 30 22:30:26.394842 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:26.394798 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f76c4-tljk6"] Oct 30 22:30:27.603797 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:27.603757 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a0667a5-c67a-46fd-8728-1fc6a950eb58" path="/var/lib/kubelet/pods/4a0667a5-c67a-46fd-8728-1fc6a950eb58/volumes" Oct 30 22:30:27.912361 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:27.912272 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:30:34.302124 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302078 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5686cb-gngr8"] Oct 30 22:30:34.302758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302631 6006 cpu_manager.go:414] "containerMap: removing container" podUID="34557f8d-a341-4f36-9914-8ec5a9af1228" containerName="cleanup" Oct 30 22:30:34.302758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302645 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34557f8d-a341-4f36-9914-8ec5a9af1228" containerName="cleanup" Oct 30 22:30:34.302758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302655 6006 cpu_manager.go:414] "containerMap: removing container" podUID="08446265-b751-4ff4-a788-488684eba3ea" containerName="cleanup" Oct 30 22:30:34.302758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302660 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08446265-b751-4ff4-a788-488684eba3ea" containerName="cleanup" Oct 30 22:30:34.302758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302671 6006 cpu_manager.go:414] "containerMap: removing container" podUID="92efe8ed-fd67-4f56-a4d0-a7ef63e8964f" containerName="cleanup" Oct 30 22:30:34.302758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302676 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92efe8ed-fd67-4f56-a4d0-a7ef63e8964f" containerName="cleanup" Oct 30 22:30:34.302758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302683 6006 cpu_manager.go:414] "containerMap: removing container" podUID="cd8cd9b2-105d-4115-a810-44e907efaba4" containerName="cleanup" Oct 30 22:30:34.302758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302689 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd8cd9b2-105d-4115-a810-44e907efaba4" containerName="cleanup" Oct 30 22:30:34.302758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302695 6006 cpu_manager.go:414] "containerMap: removing container" podUID="f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3" containerName="istio-proxy" Oct 30 22:30:34.302758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302701 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3" containerName="istio-proxy" Oct 30 22:30:34.302758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302707 6006 cpu_manager.go:414] "containerMap: removing container" podUID="34557f8d-a341-4f36-9914-8ec5a9af1228" containerName="cleanup" Oct 30 22:30:34.302758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302712 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34557f8d-a341-4f36-9914-8ec5a9af1228" containerName="cleanup" Oct 30 22:30:34.302758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302719 6006 cpu_manager.go:414] "containerMap: removing container" podUID="d9284e65-6486-487b-a207-a1b48b7ee19b" containerName="cleanup" Oct 30 22:30:34.302758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302724 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9284e65-6486-487b-a207-a1b48b7ee19b" containerName="cleanup" Oct 30 22:30:34.302758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302731 6006 cpu_manager.go:414] "containerMap: removing container" podUID="92efe8ed-fd67-4f56-a4d0-a7ef63e8964f" containerName="cleanup" Oct 30 22:30:34.302758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302736 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92efe8ed-fd67-4f56-a4d0-a7ef63e8964f" containerName="cleanup" Oct 30 22:30:34.302758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302744 6006 cpu_manager.go:414] "containerMap: removing container" podUID="cf68b963-460e-4af7-8f71-d00b8e0449fc" containerName="cleanup" Oct 30 22:30:34.302758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302749 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf68b963-460e-4af7-8f71-d00b8e0449fc" containerName="cleanup" Oct 30 22:30:34.302758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302755 6006 cpu_manager.go:414] "containerMap: removing container" podUID="fece6ef8-ca00-4666-9860-0383ee19082c" containerName="cleanup" Oct 30 22:30:34.302758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302759 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fece6ef8-ca00-4666-9860-0383ee19082c" containerName="cleanup" Oct 30 22:30:34.302758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302765 6006 cpu_manager.go:414] "containerMap: removing container" podUID="94535d2d-703e-4804-992a-474908adf7f2" containerName="cleanup" Oct 30 22:30:34.302758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302770 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94535d2d-703e-4804-992a-474908adf7f2" containerName="cleanup" Oct 30 22:30:34.302758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302777 6006 cpu_manager.go:414] "containerMap: removing container" podUID="ab56f9ed-7e2f-4a74-8dda-53df36c32f23" containerName="istio-proxy" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302781 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab56f9ed-7e2f-4a74-8dda-53df36c32f23" containerName="istio-proxy" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302790 6006 cpu_manager.go:414] "containerMap: removing container" podUID="94535d2d-703e-4804-992a-474908adf7f2" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302794 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94535d2d-703e-4804-992a-474908adf7f2" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302804 6006 cpu_manager.go:414] "containerMap: removing container" podUID="f33ebace-e652-4f07-bbce-9c334faa3948" containerName="istio-proxy" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302809 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f33ebace-e652-4f07-bbce-9c334faa3948" containerName="istio-proxy" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302816 6006 cpu_manager.go:414] "containerMap: removing container" podUID="cf68b963-460e-4af7-8f71-d00b8e0449fc" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302821 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf68b963-460e-4af7-8f71-d00b8e0449fc" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302830 6006 cpu_manager.go:414] "containerMap: removing container" podUID="34557f8d-a341-4f36-9914-8ec5a9af1228" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302835 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34557f8d-a341-4f36-9914-8ec5a9af1228" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302844 6006 cpu_manager.go:414] "containerMap: removing container" podUID="88f78b1a-a774-44f1-8a64-8786dcdcf528" containerName="istio-proxy" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302848 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88f78b1a-a774-44f1-8a64-8786dcdcf528" containerName="istio-proxy" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302854 6006 cpu_manager.go:414] "containerMap: removing container" podUID="94535d2d-703e-4804-992a-474908adf7f2" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302859 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94535d2d-703e-4804-992a-474908adf7f2" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302867 6006 cpu_manager.go:414] "containerMap: removing container" podUID="d9284e65-6486-487b-a207-a1b48b7ee19b" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302871 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9284e65-6486-487b-a207-a1b48b7ee19b" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302879 6006 cpu_manager.go:414] "containerMap: removing container" podUID="dcc289f1-3dd7-4182-bc07-8ca05b24b13b" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302883 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcc289f1-3dd7-4182-bc07-8ca05b24b13b" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302892 6006 cpu_manager.go:414] "containerMap: removing container" podUID="d9284e65-6486-487b-a207-a1b48b7ee19b" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302896 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9284e65-6486-487b-a207-a1b48b7ee19b" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302903 6006 cpu_manager.go:414] "containerMap: removing container" podUID="cd8cd9b2-105d-4115-a810-44e907efaba4" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302908 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd8cd9b2-105d-4115-a810-44e907efaba4" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302916 6006 cpu_manager.go:414] "containerMap: removing container" podUID="dcc289f1-3dd7-4182-bc07-8ca05b24b13b" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302920 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcc289f1-3dd7-4182-bc07-8ca05b24b13b" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302926 6006 cpu_manager.go:414] "containerMap: removing container" podUID="fece6ef8-ca00-4666-9860-0383ee19082c" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302930 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fece6ef8-ca00-4666-9860-0383ee19082c" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302939 6006 cpu_manager.go:414] "containerMap: removing container" podUID="08446265-b751-4ff4-a788-488684eba3ea" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.302944 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08446265-b751-4ff4-a788-488684eba3ea" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.303112 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcc289f1-3dd7-4182-bc07-8ca05b24b13b" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.303125 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="fece6ef8-ca00-4666-9860-0383ee19082c" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.303135 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="08446265-b751-4ff4-a788-488684eba3ea" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.303145 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf68b963-460e-4af7-8f71-d00b8e0449fc" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.303152 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="08446265-b751-4ff4-a788-488684eba3ea" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.303162 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcc289f1-3dd7-4182-bc07-8ca05b24b13b" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.303172 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="34557f8d-a341-4f36-9914-8ec5a9af1228" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.303180 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="08446265-b751-4ff4-a788-488684eba3ea" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.303191 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="92efe8ed-fd67-4f56-a4d0-a7ef63e8964f" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.303199 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd8cd9b2-105d-4115-a810-44e907efaba4" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.303208 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2d7d6f3-7301-4ff6-a8a9-cd2917c8cfb3" containerName="istio-proxy" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.303215 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="34557f8d-a341-4f36-9914-8ec5a9af1228" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.303259 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9284e65-6486-487b-a207-a1b48b7ee19b" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.303269 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="92efe8ed-fd67-4f56-a4d0-a7ef63e8964f" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.303277 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="fece6ef8-ca00-4666-9860-0383ee19082c" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.303285 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="94535d2d-703e-4804-992a-474908adf7f2" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.303294 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab56f9ed-7e2f-4a74-8dda-53df36c32f23" containerName="istio-proxy" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.303301 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="94535d2d-703e-4804-992a-474908adf7f2" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.303311 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="f33ebace-e652-4f07-bbce-9c334faa3948" containerName="istio-proxy" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.303320 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf68b963-460e-4af7-8f71-d00b8e0449fc" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.303330 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="88f78b1a-a774-44f1-8a64-8786dcdcf528" containerName="istio-proxy" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.303339 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="94535d2d-703e-4804-992a-474908adf7f2" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.303348 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="92efe8ed-fd67-4f56-a4d0-a7ef63e8964f" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.303359 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9284e65-6486-487b-a207-a1b48b7ee19b" containerName="cleanup" Oct 30 22:30:34.303868 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.303367 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd8cd9b2-105d-4115-a810-44e907efaba4" containerName="cleanup" Oct 30 22:30:34.310940 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.310193 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5686cb-gngr8" Oct 30 22:30:34.362005 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.361963 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5686cb-gngr8"] Oct 30 22:30:34.402720 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.402680 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5svzd\" (UniqueName: \"kubernetes.io/projected/7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9-kube-api-access-5svzd\") pod \"maas-api-revoke-keys-e2e-api-b-5686cb-gngr8\" (UID: \"7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5686cb-gngr8" Oct 30 22:30:34.402943 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.402853 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5686cb-gngr8\" (UID: \"7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5686cb-gngr8" Oct 30 22:30:34.402943 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.402914 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5686cb-gngr8\" (UID: \"7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5686cb-gngr8" Oct 30 22:30:34.451992 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.451950 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:30:34.452192 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.452001 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:30:34.504653 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.504604 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5686cb-gngr8\" (UID: \"7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5686cb-gngr8" Oct 30 22:30:34.504861 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.504702 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5686cb-gngr8\" (UID: \"7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5686cb-gngr8" Oct 30 22:30:34.504861 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.504829 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5svzd\" (UniqueName: \"kubernetes.io/projected/7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9-kube-api-access-5svzd\") pod \"maas-api-revoke-keys-e2e-api-b-5686cb-gngr8\" (UID: \"7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5686cb-gngr8" Oct 30 22:30:34.505447 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.505384 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5686cb-gngr8\" (UID: \"7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5686cb-gngr8" Oct 30 22:30:34.507424 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.507397 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5686cb-gngr8\" (UID: \"7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5686cb-gngr8" Oct 30 22:30:34.564758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.564681 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5svzd\" (UniqueName: \"kubernetes.io/projected/7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9-kube-api-access-5svzd\") pod \"maas-api-revoke-keys-e2e-api-b-5686cb-gngr8\" (UID: \"7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5686cb-gngr8" Oct 30 22:30:34.634010 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:34.633969 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5686cb-gngr8" Oct 30 22:30:35.008759 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:35.008730 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5686cb-gngr8"] Oct 30 22:30:35.011946 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:30:35.011903 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dab58ad_1dd5_46ba_beb2_b32ea5f7fce9.slice/crio-3895f6750b2b82deb6b5facd69b1c76c06491bd5374eca5ef72ee39e5c31332a WatchSource:0}: Error finding container 3895f6750b2b82deb6b5facd69b1c76c06491bd5374eca5ef72ee39e5c31332a: Status 404 returned error can't find the container with id 3895f6750b2b82deb6b5facd69b1c76c06491bd5374eca5ef72ee39e5c31332a Oct 30 22:30:35.774831 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:35.774734 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5686cb-gngr8" event={"ID":"7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9","Type":"ContainerStarted","Data":"aeb178d0a439b181855d5ad61291bca601eed05962bef15a69d4cd351e4fca81"} Oct 30 22:30:35.774831 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:35.774801 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5686cb-gngr8" event={"ID":"7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9","Type":"ContainerStarted","Data":"3895f6750b2b82deb6b5facd69b1c76c06491bd5374eca5ef72ee39e5c31332a"} Oct 30 22:30:35.889808 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:35.889749 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5686cb-gngr8" podStartSLOduration=1.889733833 podStartE2EDuration="1.889733833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:35.886285007 +0000 UTC m=+2438.438461297" watchObservedRunningTime="2026-10-30 22:30:35.889733833 +0000 UTC m=+2438.441910134" Oct 30 22:30:36.783113 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:36.783078 6006 generic.go:356] "Generic (PLEG): container finished" podID="7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9" containerID="aeb178d0a439b181855d5ad61291bca601eed05962bef15a69d4cd351e4fca81" exitCode=0 Oct 30 22:30:36.783591 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:36.783164 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5686cb-gngr8" event={"ID":"7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9","Type":"ContainerDied","Data":"aeb178d0a439b181855d5ad61291bca601eed05962bef15a69d4cd351e4fca81"} Oct 30 22:30:37.948482 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:37.948451 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5686cb-gngr8" Oct 30 22:30:38.050050 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:38.050017 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9-maas-api-openshift-ca\") pod \"7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9\" (UID: \"7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9\") " Oct 30 22:30:38.050340 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:38.050069 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9-kube-api-access-5svzd\" (UniqueName: \"kubernetes.io/projected/7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9-kube-api-access-5svzd\") pod \"7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9\" (UID: \"7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9\") " Oct 30 22:30:38.050340 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:38.050177 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9-maas-api-xks-ca\") pod \"7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9\" (UID: \"7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9\") " Oct 30 22:30:38.050470 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:38.050417 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9-maas-api-openshift-ca" pod "7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9" (UID: "7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:38.050773 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:38.050753 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:38.052757 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:38.052727 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9-maas-api-xks-ca" pod "7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9" (UID: "7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:30:38.052878 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:38.052822 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9-kube-api-access-5svzd" pod "7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9" (UID: "7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9"). InnerVolumeSpecName "kube-api-access-5svzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:38.152544 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:38.152506 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5svzd\" (UniqueName: \"kubernetes.io/projected/7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9-kube-api-access-5svzd\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:38.152544 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:38.152539 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:38.798932 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:38.798892 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5686cb-gngr8" Oct 30 22:30:38.799142 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:38.798934 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5686cb-gngr8" event={"ID":"7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9","Type":"ContainerDied","Data":"3895f6750b2b82deb6b5facd69b1c76c06491bd5374eca5ef72ee39e5c31332a"} Oct 30 22:30:38.799142 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:38.798982 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3895f6750b2b82deb6b5facd69b1c76c06491bd5374eca5ef72ee39e5c31332a" Oct 30 22:30:40.375859 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:40.375815 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:30:55.004271 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:55.000817 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r"] Oct 30 22:30:55.004271 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:55.001118 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" podUID="9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" containerName="istio-proxy" containerID="cri-o://ea90b37416b6bd56b2870a5531f4aadd22f4becb74e4a4781146eadae9cc43dc" gracePeriod=30 Oct 30 22:30:56.860970 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:56.860926 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq"] Oct 30 22:30:56.906708 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:56.906674 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-99304b-2kjsq"] Oct 30 22:30:57.605471 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:30:57.605423 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9033ff6b-3ba3-42cd-b5e5-3dc557a05acb" path="/var/lib/kubelet/pods/9033ff6b-3ba3-42cd-b5e5-3dc557a05acb/volumes" Oct 30 22:30:57.910893 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:30:57.910808 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:31:00.287529 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.287460 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:31:00.342109 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.342069 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-workload-certs\") pod \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " Oct 30 22:31:00.342321 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.342129 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-envoy\") pod \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " Oct 30 22:31:00.342321 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.342156 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-data\") pod \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " Oct 30 22:31:00.342321 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.342180 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-token\" (UniqueName: \"kubernetes.io/projected/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-token\") pod \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " Oct 30 22:31:00.342321 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.342219 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istiod-ca-cert\") pod \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " Oct 30 22:31:00.342321 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.342292 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-credential-socket\") pod \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " Oct 30 22:31:00.342611 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.342335 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-kube-api-access-x4rst\" (UniqueName: \"kubernetes.io/projected/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-kube-api-access-x4rst\") pod \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " Oct 30 22:31:00.342611 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.342395 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-workload-socket\") pod \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " Oct 30 22:31:00.342611 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.342459 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-workload-certs" pod "9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" (UID: "9c8bf63b-6ee1-48db-839f-0bd5a6c1743a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:00.342611 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.342552 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-data" pod "9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" (UID: "9c8bf63b-6ee1-48db-839f-0bd5a6c1743a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:00.342611 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.342579 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-podinfo\") pod \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\" (UID: \"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a\") " Oct 30 22:31:00.342885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.342745 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-credential-socket" pod "9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" (UID: "9c8bf63b-6ee1-48db-839f-0bd5a6c1743a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:00.342885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.342765 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-workload-socket" pod "9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" (UID: "9c8bf63b-6ee1-48db-839f-0bd5a6c1743a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:00.343034 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.343012 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istiod-ca-cert" pod "9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" (UID: "9c8bf63b-6ee1-48db-839f-0bd5a6c1743a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:00.343144 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.343128 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:00.343204 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.343151 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:00.343204 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.343165 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:00.343204 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.343180 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:00.343204 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.343194 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:00.344667 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.344629 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-envoy" pod "9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" (UID: "9c8bf63b-6ee1-48db-839f-0bd5a6c1743a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:00.344808 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.344775 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-kube-api-access-x4rst" pod "9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" (UID: "9c8bf63b-6ee1-48db-839f-0bd5a6c1743a"). InnerVolumeSpecName "kube-api-access-x4rst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:00.344875 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.344841 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-token" pod "9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" (UID: "9c8bf63b-6ee1-48db-839f-0bd5a6c1743a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:00.345019 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.344999 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-podinfo" pod "9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" (UID: "9c8bf63b-6ee1-48db-839f-0bd5a6c1743a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:00.444716 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.444671 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:00.444716 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.444711 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:00.444716 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.444720 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:00.444982 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.444731 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4rst\" (UniqueName: \"kubernetes.io/projected/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a-kube-api-access-x4rst\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:00.974590 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.974547 6006 generic.go:356] "Generic (PLEG): container finished" podID="9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" containerID="ea90b37416b6bd56b2870a5531f4aadd22f4becb74e4a4781146eadae9cc43dc" exitCode=0 Oct 30 22:31:00.974806 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.974604 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" Oct 30 22:31:00.974806 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.974660 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" event={"ID":"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a","Type":"ContainerDied","Data":"ea90b37416b6bd56b2870a5531f4aadd22f4becb74e4a4781146eadae9cc43dc"} Oct 30 22:31:00.974806 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.974739 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r" event={"ID":"9c8bf63b-6ee1-48db-839f-0bd5a6c1743a","Type":"ContainerDied","Data":"5fb8a868e85b3027e014567a5b580bd98268aa51d7f4fec24447b3660d5f443f"} Oct 30 22:31:00.974806 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.974770 6006 scope.go:122] "RemoveContainer" containerID="ea90b37416b6bd56b2870a5531f4aadd22f4becb74e4a4781146eadae9cc43dc" Oct 30 22:31:00.990074 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.990045 6006 scope.go:122] "RemoveContainer" containerID="ea90b37416b6bd56b2870a5531f4aadd22f4becb74e4a4781146eadae9cc43dc" Oct 30 22:31:00.990487 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:31:00.990455 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea90b37416b6bd56b2870a5531f4aadd22f4becb74e4a4781146eadae9cc43dc\": container with ID starting with ea90b37416b6bd56b2870a5531f4aadd22f4becb74e4a4781146eadae9cc43dc not found: ID does not exist" containerID="ea90b37416b6bd56b2870a5531f4aadd22f4becb74e4a4781146eadae9cc43dc" Oct 30 22:31:00.990603 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:00.990501 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea90b37416b6bd56b2870a5531f4aadd22f4becb74e4a4781146eadae9cc43dc"} err="failed to get container status \"ea90b37416b6bd56b2870a5531f4aadd22f4becb74e4a4781146eadae9cc43dc\": rpc error: code = NotFound desc = could not find container \"ea90b37416b6bd56b2870a5531f4aadd22f4becb74e4a4781146eadae9cc43dc\": container with ID starting with ea90b37416b6bd56b2870a5531f4aadd22f4becb74e4a4781146eadae9cc43dc not found: ID does not exist" Oct 30 22:31:01.165392 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:01.163337 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r"] Oct 30 22:31:01.167340 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:01.167300 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-bqd4r"] Oct 30 22:31:01.633314 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:01.633274 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" path="/var/lib/kubelet/pods/9c8bf63b-6ee1-48db-839f-0bd5a6c1743a/volumes" Oct 30 22:31:02.758506 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:02.758471 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs"] Oct 30 22:31:02.804671 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:02.804633 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-65e59f-f8cxs"] Oct 30 22:31:03.605802 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:03.605763 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f8aeb41-2899-4704-b20f-ba1ba69e1b74" path="/var/lib/kubelet/pods/4f8aeb41-2899-4704-b20f-ba1ba69e1b74/volumes" Oct 30 22:31:05.763452 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.763416 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc"] Oct 30 22:31:05.763967 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.763951 6006 cpu_manager.go:414] "containerMap: removing container" podUID="cd8cd9b2-105d-4115-a810-44e907efaba4" containerName="cleanup" Oct 30 22:31:05.764012 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.763969 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd8cd9b2-105d-4115-a810-44e907efaba4" containerName="cleanup" Oct 30 22:31:05.764012 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.763991 6006 cpu_manager.go:414] "containerMap: removing container" podUID="92efe8ed-fd67-4f56-a4d0-a7ef63e8964f" containerName="cleanup" Oct 30 22:31:05.764012 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.763999 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92efe8ed-fd67-4f56-a4d0-a7ef63e8964f" containerName="cleanup" Oct 30 22:31:05.764121 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.764025 6006 cpu_manager.go:414] "containerMap: removing container" podUID="cf68b963-460e-4af7-8f71-d00b8e0449fc" containerName="cleanup" Oct 30 22:31:05.764121 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.764034 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf68b963-460e-4af7-8f71-d00b8e0449fc" containerName="cleanup" Oct 30 22:31:05.764121 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.764068 6006 cpu_manager.go:414] "containerMap: removing container" podUID="9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" containerName="istio-proxy" Oct 30 22:31:05.764121 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.764077 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" containerName="istio-proxy" Oct 30 22:31:05.764121 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.764092 6006 cpu_manager.go:414] "containerMap: removing container" podUID="dcc289f1-3dd7-4182-bc07-8ca05b24b13b" containerName="cleanup" Oct 30 22:31:05.764121 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.764106 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcc289f1-3dd7-4182-bc07-8ca05b24b13b" containerName="cleanup" Oct 30 22:31:05.764357 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.764127 6006 cpu_manager.go:414] "containerMap: removing container" podUID="fece6ef8-ca00-4666-9860-0383ee19082c" containerName="cleanup" Oct 30 22:31:05.764357 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.764136 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fece6ef8-ca00-4666-9860-0383ee19082c" containerName="cleanup" Oct 30 22:31:05.764357 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.764170 6006 cpu_manager.go:414] "containerMap: removing container" podUID="7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9" containerName="revoke-keys" Oct 30 22:31:05.764357 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.764178 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9" containerName="revoke-keys" Oct 30 22:31:05.764357 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.764197 6006 cpu_manager.go:414] "containerMap: removing container" podUID="08446265-b751-4ff4-a788-488684eba3ea" containerName="cleanup" Oct 30 22:31:05.764357 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.764207 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08446265-b751-4ff4-a788-488684eba3ea" containerName="cleanup" Oct 30 22:31:05.764563 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.764368 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9284e65-6486-487b-a207-a1b48b7ee19b" containerName="cleanup" Oct 30 22:31:05.764563 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.764382 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c8bf63b-6ee1-48db-839f-0bd5a6c1743a" containerName="istio-proxy" Oct 30 22:31:05.764563 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.764411 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf68b963-460e-4af7-8f71-d00b8e0449fc" containerName="cleanup" Oct 30 22:31:05.764563 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.764419 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcc289f1-3dd7-4182-bc07-8ca05b24b13b" containerName="cleanup" Oct 30 22:31:05.764563 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.764426 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9" containerName="revoke-keys" Oct 30 22:31:05.764563 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.764432 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd8cd9b2-105d-4115-a810-44e907efaba4" containerName="cleanup" Oct 30 22:31:05.764563 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.764439 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="34557f8d-a341-4f36-9914-8ec5a9af1228" containerName="cleanup" Oct 30 22:31:05.764563 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.764447 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="fece6ef8-ca00-4666-9860-0383ee19082c" containerName="cleanup" Oct 30 22:31:05.769463 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.769438 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc" Oct 30 22:31:05.817902 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.817865 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrr8j\" (UniqueName: \"kubernetes.io/projected/15614cea-f680-44fb-bc46-7d632d5c6770-kube-api-access-vrr8j\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc\" (UID: \"15614cea-f680-44fb-bc46-7d632d5c6770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc" Oct 30 22:31:05.818118 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.817987 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15614cea-f680-44fb-bc46-7d632d5c6770-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc\" (UID: \"15614cea-f680-44fb-bc46-7d632d5c6770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc" Oct 30 22:31:05.818118 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.818030 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15614cea-f680-44fb-bc46-7d632d5c6770-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc\" (UID: \"15614cea-f680-44fb-bc46-7d632d5c6770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc" Oct 30 22:31:05.819761 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.819722 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:31:05.866304 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.866220 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:31:05.868904 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.868870 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc"] Oct 30 22:31:05.919080 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.919037 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrr8j\" (UniqueName: \"kubernetes.io/projected/15614cea-f680-44fb-bc46-7d632d5c6770-kube-api-access-vrr8j\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc\" (UID: \"15614cea-f680-44fb-bc46-7d632d5c6770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc" Oct 30 22:31:05.919328 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.919130 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15614cea-f680-44fb-bc46-7d632d5c6770-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc\" (UID: \"15614cea-f680-44fb-bc46-7d632d5c6770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc" Oct 30 22:31:05.919328 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.919173 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15614cea-f680-44fb-bc46-7d632d5c6770-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc\" (UID: \"15614cea-f680-44fb-bc46-7d632d5c6770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc" Oct 30 22:31:05.920026 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.919993 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15614cea-f680-44fb-bc46-7d632d5c6770-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc\" (UID: \"15614cea-f680-44fb-bc46-7d632d5c6770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc" Oct 30 22:31:05.922173 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.922146 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15614cea-f680-44fb-bc46-7d632d5c6770-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc\" (UID: \"15614cea-f680-44fb-bc46-7d632d5c6770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc" Oct 30 22:31:05.977324 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:05.977288 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrr8j\" (UniqueName: \"kubernetes.io/projected/15614cea-f680-44fb-bc46-7d632d5c6770-kube-api-access-vrr8j\") pod \"maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc\" (UID: \"15614cea-f680-44fb-bc46-7d632d5c6770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc" Oct 30 22:31:06.082877 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:06.082839 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc" Oct 30 22:31:06.466140 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:06.466112 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc"] Oct 30 22:31:06.468560 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:31:06.468530 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15614cea_f680_44fb_bc46_7d632d5c6770.slice/crio-bcb258b775052287f235d9aa53ed19207263f69d65dd95b27e81f5b85a58a5c0 WatchSource:0}: Error finding container bcb258b775052287f235d9aa53ed19207263f69d65dd95b27e81f5b85a58a5c0: Status 404 returned error can't find the container with id bcb258b775052287f235d9aa53ed19207263f69d65dd95b27e81f5b85a58a5c0 Oct 30 22:31:07.023493 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:07.023396 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc" event={"ID":"15614cea-f680-44fb-bc46-7d632d5c6770","Type":"ContainerStarted","Data":"0543281f09eab4f1e5e9d7e83ff3f62aa08a6b2322355f8d3c2593aeac51897d"} Oct 30 22:31:07.023493 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:07.023457 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc" event={"ID":"15614cea-f680-44fb-bc46-7d632d5c6770","Type":"ContainerStarted","Data":"bcb258b775052287f235d9aa53ed19207263f69d65dd95b27e81f5b85a58a5c0"} Oct 30 22:31:07.148441 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:07.148364 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc" podStartSLOduration=2.148345805 podStartE2EDuration="2.148345805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:07.146202509 +0000 UTC m=+2469.698378826" watchObservedRunningTime="2026-10-30 22:31:07.148345805 +0000 UTC m=+2469.700522107" Oct 30 22:31:08.033418 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:08.033383 6006 generic.go:356] "Generic (PLEG): container finished" podID="15614cea-f680-44fb-bc46-7d632d5c6770" containerID="0543281f09eab4f1e5e9d7e83ff3f62aa08a6b2322355f8d3c2593aeac51897d" exitCode=0 Oct 30 22:31:08.033838 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:08.033476 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc" event={"ID":"15614cea-f680-44fb-bc46-7d632d5c6770","Type":"ContainerDied","Data":"0543281f09eab4f1e5e9d7e83ff3f62aa08a6b2322355f8d3c2593aeac51897d"} Oct 30 22:31:09.194234 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:09.194210 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc" Oct 30 22:31:09.274721 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:09.274680 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15614cea-f680-44fb-bc46-7d632d5c6770-kube-api-access-vrr8j\" (UniqueName: \"kubernetes.io/projected/15614cea-f680-44fb-bc46-7d632d5c6770-kube-api-access-vrr8j\") pod \"15614cea-f680-44fb-bc46-7d632d5c6770\" (UID: \"15614cea-f680-44fb-bc46-7d632d5c6770\") " Oct 30 22:31:09.274938 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:09.274757 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15614cea-f680-44fb-bc46-7d632d5c6770-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15614cea-f680-44fb-bc46-7d632d5c6770-maas-api-openshift-ca\") pod \"15614cea-f680-44fb-bc46-7d632d5c6770\" (UID: \"15614cea-f680-44fb-bc46-7d632d5c6770\") " Oct 30 22:31:09.275015 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:09.274947 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15614cea-f680-44fb-bc46-7d632d5c6770-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15614cea-f680-44fb-bc46-7d632d5c6770-maas-api-xks-ca\") pod \"15614cea-f680-44fb-bc46-7d632d5c6770\" (UID: \"15614cea-f680-44fb-bc46-7d632d5c6770\") " Oct 30 22:31:09.275165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:09.275133 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15614cea-f680-44fb-bc46-7d632d5c6770-maas-api-openshift-ca" pod "15614cea-f680-44fb-bc46-7d632d5c6770" (UID: "15614cea-f680-44fb-bc46-7d632d5c6770"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:09.275385 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:09.275361 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15614cea-f680-44fb-bc46-7d632d5c6770-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:09.277329 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:09.277301 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15614cea-f680-44fb-bc46-7d632d5c6770-kube-api-access-vrr8j" pod "15614cea-f680-44fb-bc46-7d632d5c6770" (UID: "15614cea-f680-44fb-bc46-7d632d5c6770"). InnerVolumeSpecName "kube-api-access-vrr8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:09.277447 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:09.277359 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15614cea-f680-44fb-bc46-7d632d5c6770-maas-api-xks-ca" pod "15614cea-f680-44fb-bc46-7d632d5c6770" (UID: "15614cea-f680-44fb-bc46-7d632d5c6770"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:09.376556 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:09.376469 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrr8j\" (UniqueName: \"kubernetes.io/projected/15614cea-f680-44fb-bc46-7d632d5c6770-kube-api-access-vrr8j\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:09.376556 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:09.376504 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15614cea-f680-44fb-bc46-7d632d5c6770-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:10.049351 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:10.049312 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc" Oct 30 22:31:10.049537 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:10.049366 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc" event={"ID":"15614cea-f680-44fb-bc46-7d632d5c6770","Type":"ContainerDied","Data":"bcb258b775052287f235d9aa53ed19207263f69d65dd95b27e81f5b85a58a5c0"} Oct 30 22:31:10.049537 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:10.049414 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcb258b775052287f235d9aa53ed19207263f69d65dd95b27e81f5b85a58a5c0" Oct 30 22:31:10.226182 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:10.226142 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt"] Oct 30 22:31:10.266746 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:10.266705 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g"] Oct 30 22:31:10.282166 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:10.282123 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss"] Oct 30 22:31:21.523594 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.523544 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4ce8ef7-kx9vm"] Oct 30 22:31:21.524349 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.524327 6006 cpu_manager.go:414] "containerMap: removing container" podUID="15614cea-f680-44fb-bc46-7d632d5c6770" containerName="revoke-keys" Oct 30 22:31:21.524428 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.524352 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15614cea-f680-44fb-bc46-7d632d5c6770" containerName="revoke-keys" Oct 30 22:31:21.524562 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.524544 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="15614cea-f680-44fb-bc46-7d632d5c6770" containerName="revoke-keys" Oct 30 22:31:21.528648 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.528621 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ce8ef7-kx9vm" Oct 30 22:31:21.584367 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.584331 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4ce8ef7-kx9vm"] Oct 30 22:31:21.627429 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.627395 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:31:21.628808 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.628779 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:31:21.635771 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.635732 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260a3f69-701a-4ff2-a2cd-769957f80522-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4ce8ef7-kx9vm\" (UID: \"260a3f69-701a-4ff2-a2cd-769957f80522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ce8ef7-kx9vm" Oct 30 22:31:21.635934 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.635854 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krvm8\" (UniqueName: \"kubernetes.io/projected/260a3f69-701a-4ff2-a2cd-769957f80522-kube-api-access-krvm8\") pod \"maas-api-revoke-sub-b4ce8ef7-kx9vm\" (UID: \"260a3f69-701a-4ff2-a2cd-769957f80522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ce8ef7-kx9vm" Oct 30 22:31:21.636021 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.636003 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260a3f69-701a-4ff2-a2cd-769957f80522-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4ce8ef7-kx9vm\" (UID: \"260a3f69-701a-4ff2-a2cd-769957f80522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ce8ef7-kx9vm" Oct 30 22:31:21.637825 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.637798 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd3d021e-97bzr"] Oct 30 22:31:21.648823 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.648788 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3d021e-97bzr" Oct 30 22:31:21.690266 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.690222 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd3d021e-97bzr"] Oct 30 22:31:21.737424 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.737383 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krvm8\" (UniqueName: \"kubernetes.io/projected/260a3f69-701a-4ff2-a2cd-769957f80522-kube-api-access-krvm8\") pod \"maas-api-revoke-sub-b4ce8ef7-kx9vm\" (UID: \"260a3f69-701a-4ff2-a2cd-769957f80522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ce8ef7-kx9vm" Oct 30 22:31:21.737637 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.737511 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260a3f69-701a-4ff2-a2cd-769957f80522-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4ce8ef7-kx9vm\" (UID: \"260a3f69-701a-4ff2-a2cd-769957f80522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ce8ef7-kx9vm" Oct 30 22:31:21.737637 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.737584 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d7js\" (UniqueName: \"kubernetes.io/projected/c8c456dc-1d23-456b-b272-1431f457b971-kube-api-access-4d7js\") pod \"maas-api-revoke-sub-fd3d021e-97bzr\" (UID: \"c8c456dc-1d23-456b-b272-1431f457b971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3d021e-97bzr" Oct 30 22:31:21.737818 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.737679 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260a3f69-701a-4ff2-a2cd-769957f80522-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4ce8ef7-kx9vm\" (UID: \"260a3f69-701a-4ff2-a2cd-769957f80522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ce8ef7-kx9vm" Oct 30 22:31:21.737818 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.737735 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c456dc-1d23-456b-b272-1431f457b971-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd3d021e-97bzr\" (UID: \"c8c456dc-1d23-456b-b272-1431f457b971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3d021e-97bzr" Oct 30 22:31:21.737932 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.737854 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c456dc-1d23-456b-b272-1431f457b971-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd3d021e-97bzr\" (UID: \"c8c456dc-1d23-456b-b272-1431f457b971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3d021e-97bzr" Oct 30 22:31:21.738466 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.738443 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260a3f69-701a-4ff2-a2cd-769957f80522-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4ce8ef7-kx9vm\" (UID: \"260a3f69-701a-4ff2-a2cd-769957f80522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ce8ef7-kx9vm" Oct 30 22:31:21.740348 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.740326 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260a3f69-701a-4ff2-a2cd-769957f80522-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4ce8ef7-kx9vm\" (UID: \"260a3f69-701a-4ff2-a2cd-769957f80522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ce8ef7-kx9vm" Oct 30 22:31:21.796012 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.795977 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krvm8\" (UniqueName: \"kubernetes.io/projected/260a3f69-701a-4ff2-a2cd-769957f80522-kube-api-access-krvm8\") pod \"maas-api-revoke-sub-b4ce8ef7-kx9vm\" (UID: \"260a3f69-701a-4ff2-a2cd-769957f80522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ce8ef7-kx9vm" Oct 30 22:31:21.839505 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.839466 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d7js\" (UniqueName: \"kubernetes.io/projected/c8c456dc-1d23-456b-b272-1431f457b971-kube-api-access-4d7js\") pod \"maas-api-revoke-sub-fd3d021e-97bzr\" (UID: \"c8c456dc-1d23-456b-b272-1431f457b971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3d021e-97bzr" Oct 30 22:31:21.839706 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.839563 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c456dc-1d23-456b-b272-1431f457b971-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd3d021e-97bzr\" (UID: \"c8c456dc-1d23-456b-b272-1431f457b971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3d021e-97bzr" Oct 30 22:31:21.839706 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.839617 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c456dc-1d23-456b-b272-1431f457b971-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd3d021e-97bzr\" (UID: \"c8c456dc-1d23-456b-b272-1431f457b971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3d021e-97bzr" Oct 30 22:31:21.840369 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.840341 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c456dc-1d23-456b-b272-1431f457b971-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd3d021e-97bzr\" (UID: \"c8c456dc-1d23-456b-b272-1431f457b971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3d021e-97bzr" Oct 30 22:31:21.842031 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.842003 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c456dc-1d23-456b-b272-1431f457b971-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd3d021e-97bzr\" (UID: \"c8c456dc-1d23-456b-b272-1431f457b971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3d021e-97bzr" Oct 30 22:31:21.848067 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.848037 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ce8ef7-kx9vm" Oct 30 22:31:21.898889 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.898856 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d7js\" (UniqueName: \"kubernetes.io/projected/c8c456dc-1d23-456b-b272-1431f457b971-kube-api-access-4d7js\") pod \"maas-api-revoke-sub-fd3d021e-97bzr\" (UID: \"c8c456dc-1d23-456b-b272-1431f457b971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3d021e-97bzr" Oct 30 22:31:21.962437 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:21.962408 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3d021e-97bzr" Oct 30 22:31:22.226348 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:31:22.226312 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260a3f69_701a_4ff2_a2cd_769957f80522.slice/crio-9e702f7e6186d4afb95ffd88f75f9ab073b97bf77b6cd051f1cff53734bc7d27 WatchSource:0}: Error finding container 9e702f7e6186d4afb95ffd88f75f9ab073b97bf77b6cd051f1cff53734bc7d27: Status 404 returned error can't find the container with id 9e702f7e6186d4afb95ffd88f75f9ab073b97bf77b6cd051f1cff53734bc7d27 Oct 30 22:31:22.229322 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:22.229288 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4ce8ef7-kx9vm"] Oct 30 22:31:22.342892 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:22.342858 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd3d021e-97bzr"] Oct 30 22:31:22.343753 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:31:22.343726 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c456dc_1d23_456b_b272_1431f457b971.slice/crio-101008875df81859a5630c6d977e8ab8c7d16dfc81cb55f85cf9820eb48bc8cb WatchSource:0}: Error finding container 101008875df81859a5630c6d977e8ab8c7d16dfc81cb55f85cf9820eb48bc8cb: Status 404 returned error can't find the container with id 101008875df81859a5630c6d977e8ab8c7d16dfc81cb55f85cf9820eb48bc8cb Oct 30 22:31:23.142954 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:23.142910 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3d021e-97bzr" event={"ID":"c8c456dc-1d23-456b-b272-1431f457b971","Type":"ContainerStarted","Data":"618d9285f2336da155626be28220ec8dcc94a78206a2e86751a3c4ea7910daa2"} Oct 30 22:31:23.143425 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:23.142978 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3d021e-97bzr" event={"ID":"c8c456dc-1d23-456b-b272-1431f457b971","Type":"ContainerStarted","Data":"101008875df81859a5630c6d977e8ab8c7d16dfc81cb55f85cf9820eb48bc8cb"} Oct 30 22:31:23.145099 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:23.145070 6006 generic.go:356] "Generic (PLEG): container finished" podID="260a3f69-701a-4ff2-a2cd-769957f80522" containerID="f42cc628c94ec5d9decfa8291ba1936e5145410ab078678466660ce9e218a7bc" exitCode=0 Oct 30 22:31:23.145223 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:23.145151 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ce8ef7-kx9vm" event={"ID":"260a3f69-701a-4ff2-a2cd-769957f80522","Type":"ContainerDied","Data":"f42cc628c94ec5d9decfa8291ba1936e5145410ab078678466660ce9e218a7bc"} Oct 30 22:31:23.145223 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:23.145205 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ce8ef7-kx9vm" event={"ID":"260a3f69-701a-4ff2-a2cd-769957f80522","Type":"ContainerStarted","Data":"9e702f7e6186d4afb95ffd88f75f9ab073b97bf77b6cd051f1cff53734bc7d27"} Oct 30 22:31:23.256484 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:23.256426 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3d021e-97bzr" podStartSLOduration=2.256410962 podStartE2EDuration="2.256410962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:23.254572553 +0000 UTC m=+2485.806748853" watchObservedRunningTime="2026-10-30 22:31:23.256410962 +0000 UTC m=+2485.808587261" Oct 30 22:31:24.154906 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:24.154842 6006 generic.go:356] "Generic (PLEG): container finished" podID="c8c456dc-1d23-456b-b272-1431f457b971" containerID="618d9285f2336da155626be28220ec8dcc94a78206a2e86751a3c4ea7910daa2" exitCode=0 Oct 30 22:31:24.155460 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:24.155010 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3d021e-97bzr" event={"ID":"c8c456dc-1d23-456b-b272-1431f457b971","Type":"ContainerDied","Data":"618d9285f2336da155626be28220ec8dcc94a78206a2e86751a3c4ea7910daa2"} Oct 30 22:31:24.384378 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:24.384347 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ce8ef7-kx9vm" Oct 30 22:31:24.503118 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:24.503017 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/260a3f69-701a-4ff2-a2cd-769957f80522-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260a3f69-701a-4ff2-a2cd-769957f80522-maas-api-openshift-ca\") pod \"260a3f69-701a-4ff2-a2cd-769957f80522\" (UID: \"260a3f69-701a-4ff2-a2cd-769957f80522\") " Oct 30 22:31:24.503340 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:24.503206 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/260a3f69-701a-4ff2-a2cd-769957f80522-kube-api-access-krvm8\" (UniqueName: \"kubernetes.io/projected/260a3f69-701a-4ff2-a2cd-769957f80522-kube-api-access-krvm8\") pod \"260a3f69-701a-4ff2-a2cd-769957f80522\" (UID: \"260a3f69-701a-4ff2-a2cd-769957f80522\") " Oct 30 22:31:24.503408 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:24.503333 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/260a3f69-701a-4ff2-a2cd-769957f80522-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260a3f69-701a-4ff2-a2cd-769957f80522-maas-api-xks-ca\") pod \"260a3f69-701a-4ff2-a2cd-769957f80522\" (UID: \"260a3f69-701a-4ff2-a2cd-769957f80522\") " Oct 30 22:31:24.503971 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:24.503940 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/260a3f69-701a-4ff2-a2cd-769957f80522-maas-api-openshift-ca" pod "260a3f69-701a-4ff2-a2cd-769957f80522" (UID: "260a3f69-701a-4ff2-a2cd-769957f80522"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:24.506762 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:24.506732 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/260a3f69-701a-4ff2-a2cd-769957f80522-maas-api-xks-ca" pod "260a3f69-701a-4ff2-a2cd-769957f80522" (UID: "260a3f69-701a-4ff2-a2cd-769957f80522"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:24.523899 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:24.521406 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260a3f69-701a-4ff2-a2cd-769957f80522-kube-api-access-krvm8" pod "260a3f69-701a-4ff2-a2cd-769957f80522" (UID: "260a3f69-701a-4ff2-a2cd-769957f80522"). InnerVolumeSpecName "kube-api-access-krvm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:24.604735 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:24.604686 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krvm8\" (UniqueName: \"kubernetes.io/projected/260a3f69-701a-4ff2-a2cd-769957f80522-kube-api-access-krvm8\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:24.604735 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:24.604720 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260a3f69-701a-4ff2-a2cd-769957f80522-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:24.604735 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:24.604730 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260a3f69-701a-4ff2-a2cd-769957f80522-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:25.163151 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.163119 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ce8ef7-kx9vm" Oct 30 22:31:25.163681 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.163159 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4ce8ef7-kx9vm" event={"ID":"260a3f69-701a-4ff2-a2cd-769957f80522","Type":"ContainerDied","Data":"9e702f7e6186d4afb95ffd88f75f9ab073b97bf77b6cd051f1cff53734bc7d27"} Oct 30 22:31:25.163681 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.163215 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e702f7e6186d4afb95ffd88f75f9ab073b97bf77b6cd051f1cff53734bc7d27" Oct 30 22:31:25.249756 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.249701 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerName="main" containerID="cri-o://4b183b869e530fb82a87326affa9db7c50a8780269d475c2c6f6375d9d43ff5d" gracePeriod=45 Oct 30 22:31:25.289006 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.288961 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" containerID="cri-o://b271c93432210844c022fbe220ab3c64946294a1fbb5338ae2eb145ec5b07e9f" gracePeriod=45 Oct 30 22:31:25.303882 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.303844 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" containerID="cri-o://1c9a8e6cf55ecbd8b352c708dcb5dfb9ce34d089ea5963246c051ee537cba547" gracePeriod=45 Oct 30 22:31:25.413294 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.412543 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": dial tcp 10.132.0.64:8000: connect: connection refused" Oct 30 22:31:25.446879 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.446839 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3d021e-97bzr" Oct 30 22:31:25.523274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.520559 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8c456dc-1d23-456b-b272-1431f457b971-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c456dc-1d23-456b-b272-1431f457b971-maas-api-xks-ca\") pod \"c8c456dc-1d23-456b-b272-1431f457b971\" (UID: \"c8c456dc-1d23-456b-b272-1431f457b971\") " Oct 30 22:31:25.523274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.520639 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8c456dc-1d23-456b-b272-1431f457b971-kube-api-access-4d7js\" (UniqueName: \"kubernetes.io/projected/c8c456dc-1d23-456b-b272-1431f457b971-kube-api-access-4d7js\") pod \"c8c456dc-1d23-456b-b272-1431f457b971\" (UID: \"c8c456dc-1d23-456b-b272-1431f457b971\") " Oct 30 22:31:25.523274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.520680 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8c456dc-1d23-456b-b272-1431f457b971-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c456dc-1d23-456b-b272-1431f457b971-maas-api-openshift-ca\") pod \"c8c456dc-1d23-456b-b272-1431f457b971\" (UID: \"c8c456dc-1d23-456b-b272-1431f457b971\") " Oct 30 22:31:25.523274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.521629 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c456dc-1d23-456b-b272-1431f457b971-maas-api-openshift-ca" pod "c8c456dc-1d23-456b-b272-1431f457b971" (UID: "c8c456dc-1d23-456b-b272-1431f457b971"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:25.534685 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.534630 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8c456dc-1d23-456b-b272-1431f457b971-maas-api-xks-ca" pod "c8c456dc-1d23-456b-b272-1431f457b971" (UID: "c8c456dc-1d23-456b-b272-1431f457b971"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:25.535035 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.534984 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c456dc-1d23-456b-b272-1431f457b971-kube-api-access-4d7js" pod "c8c456dc-1d23-456b-b272-1431f457b971" (UID: "c8c456dc-1d23-456b-b272-1431f457b971"). InnerVolumeSpecName "kube-api-access-4d7js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:25.589200 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.589173 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:31:25.623192 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.623067 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c456dc-1d23-456b-b272-1431f457b971-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:25.623446 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.623424 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d7js\" (UniqueName: \"kubernetes.io/projected/c8c456dc-1d23-456b-b272-1431f457b971-kube-api-access-4d7js\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:25.623552 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.623541 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c456dc-1d23-456b-b272-1431f457b971-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:25.680062 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.680026 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:31:25.724660 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.724625 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-home\" (UniqueName: \"kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-home\") pod \"4e5d742b-c020-45ff-bd02-76663b0bb04a\" (UID: \"4e5d742b-c020-45ff-bd02-76663b0bb04a\") " Oct 30 22:31:25.724818 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.724756 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-tmp-dir\") pod \"4e5d742b-c020-45ff-bd02-76663b0bb04a\" (UID: \"4e5d742b-c020-45ff-bd02-76663b0bb04a\") " Oct 30 22:31:25.724818 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.724790 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e5d742b-c020-45ff-bd02-76663b0bb04a-kube-api-access-79m9b\" (UniqueName: \"kubernetes.io/projected/4e5d742b-c020-45ff-bd02-76663b0bb04a-kube-api-access-79m9b\") pod \"4e5d742b-c020-45ff-bd02-76663b0bb04a\" (UID: \"4e5d742b-c020-45ff-bd02-76663b0bb04a\") " Oct 30 22:31:25.724890 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.724851 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e5d742b-c020-45ff-bd02-76663b0bb04a-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e5d742b-c020-45ff-bd02-76663b0bb04a-tls-certs\") pod \"4e5d742b-c020-45ff-bd02-76663b0bb04a\" (UID: \"4e5d742b-c020-45ff-bd02-76663b0bb04a\") " Oct 30 22:31:25.724929 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.724908 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-model-cache\") pod \"4e5d742b-c020-45ff-bd02-76663b0bb04a\" (UID: \"4e5d742b-c020-45ff-bd02-76663b0bb04a\") " Oct 30 22:31:25.724929 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.724927 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-dshm\") pod \"4e5d742b-c020-45ff-bd02-76663b0bb04a\" (UID: \"4e5d742b-c020-45ff-bd02-76663b0bb04a\") " Oct 30 22:31:25.725703 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.725680 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-home" pod "4e5d742b-c020-45ff-bd02-76663b0bb04a" (UID: "4e5d742b-c020-45ff-bd02-76663b0bb04a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:25.725827 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.725813 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-tmp-dir" pod "4e5d742b-c020-45ff-bd02-76663b0bb04a" (UID: "4e5d742b-c020-45ff-bd02-76663b0bb04a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:25.726189 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.726167 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-model-cache" pod "4e5d742b-c020-45ff-bd02-76663b0bb04a" (UID: "4e5d742b-c020-45ff-bd02-76663b0bb04a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:25.728357 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.728324 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e5d742b-c020-45ff-bd02-76663b0bb04a-tls-certs" pod "4e5d742b-c020-45ff-bd02-76663b0bb04a" (UID: "4e5d742b-c020-45ff-bd02-76663b0bb04a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:25.729639 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.729609 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5d742b-c020-45ff-bd02-76663b0bb04a-kube-api-access-79m9b" pod "4e5d742b-c020-45ff-bd02-76663b0bb04a" (UID: "4e5d742b-c020-45ff-bd02-76663b0bb04a"). InnerVolumeSpecName "kube-api-access-79m9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:25.731579 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.731549 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-dshm" pod "4e5d742b-c020-45ff-bd02-76663b0bb04a" (UID: "4e5d742b-c020-45ff-bd02-76663b0bb04a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:25.749462 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.749428 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:31:25.826141 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.826106 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-dshm\") pod \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\" (UID: \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\") " Oct 30 22:31:25.826363 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.826167 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-home\" (UniqueName: \"kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-home\") pod \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\" (UID: \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\") " Oct 30 22:31:25.826363 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.826202 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-home\" (UniqueName: \"kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-home\") pod \"49f28c58-bbe2-4110-8ac1-42e629c8be80\" (UID: \"49f28c58-bbe2-4110-8ac1-42e629c8be80\") " Oct 30 22:31:25.826363 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.826266 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-tmp-dir\") pod \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\" (UID: \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\") " Oct 30 22:31:25.826363 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.826299 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-tmp-dir\") pod \"49f28c58-bbe2-4110-8ac1-42e629c8be80\" (UID: \"49f28c58-bbe2-4110-8ac1-42e629c8be80\") " Oct 30 22:31:25.826608 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.826435 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-dshm\" (UniqueName: \"kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-dshm\") pod \"49f28c58-bbe2-4110-8ac1-42e629c8be80\" (UID: \"49f28c58-bbe2-4110-8ac1-42e629c8be80\") " Oct 30 22:31:25.826608 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.826463 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49f28c58-bbe2-4110-8ac1-42e629c8be80-tls-certs\" (UniqueName: \"kubernetes.io/secret/49f28c58-bbe2-4110-8ac1-42e629c8be80-tls-certs\") pod \"49f28c58-bbe2-4110-8ac1-42e629c8be80\" (UID: \"49f28c58-bbe2-4110-8ac1-42e629c8be80\") " Oct 30 22:31:25.826608 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.826546 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21c2b5d2-ef4b-4b68-af47-d550a11385d7-tls-certs\" (UniqueName: \"kubernetes.io/secret/21c2b5d2-ef4b-4b68-af47-d550a11385d7-tls-certs\") pod \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\" (UID: \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\") " Oct 30 22:31:25.826608 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.826582 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21c2b5d2-ef4b-4b68-af47-d550a11385d7-kube-api-access-4lqgs\" (UniqueName: \"kubernetes.io/projected/21c2b5d2-ef4b-4b68-af47-d550a11385d7-kube-api-access-4lqgs\") pod \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\" (UID: \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\") " Oct 30 22:31:25.826810 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.826632 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-model-cache\") pod \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\" (UID: \"21c2b5d2-ef4b-4b68-af47-d550a11385d7\") " Oct 30 22:31:25.826810 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.826631 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-tmp-dir" pod "49f28c58-bbe2-4110-8ac1-42e629c8be80" (UID: "49f28c58-bbe2-4110-8ac1-42e629c8be80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:25.826810 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.826717 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-home" pod "49f28c58-bbe2-4110-8ac1-42e629c8be80" (UID: "49f28c58-bbe2-4110-8ac1-42e629c8be80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:25.837755 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.827119 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-model-cache\") pod \"49f28c58-bbe2-4110-8ac1-42e629c8be80\" (UID: \"49f28c58-bbe2-4110-8ac1-42e629c8be80\") " Oct 30 22:31:25.837755 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.827166 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49f28c58-bbe2-4110-8ac1-42e629c8be80-kube-api-access-7gv62\" (UniqueName: \"kubernetes.io/projected/49f28c58-bbe2-4110-8ac1-42e629c8be80-kube-api-access-7gv62\") pod \"49f28c58-bbe2-4110-8ac1-42e629c8be80\" (UID: \"49f28c58-bbe2-4110-8ac1-42e629c8be80\") " Oct 30 22:31:25.837755 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.828011 6006 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-tmp-dir\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:25.837755 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.828041 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79m9b\" (UniqueName: \"kubernetes.io/projected/4e5d742b-c020-45ff-bd02-76663b0bb04a-kube-api-access-79m9b\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:25.837755 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.828058 6006 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e5d742b-c020-45ff-bd02-76663b0bb04a-tls-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:25.837755 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.828074 6006 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-model-cache\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:25.837755 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.828086 6006 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-dshm\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:25.837755 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.828107 6006 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e5d742b-c020-45ff-bd02-76663b0bb04a-home\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:25.837755 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.828120 6006 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-home\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:25.837755 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.828134 6006 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-tmp-dir\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:25.837755 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.829378 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-home" pod "21c2b5d2-ef4b-4b68-af47-d550a11385d7" (UID: "21c2b5d2-ef4b-4b68-af47-d550a11385d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:25.837755 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.830637 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-tmp-dir" pod "21c2b5d2-ef4b-4b68-af47-d550a11385d7" (UID: "21c2b5d2-ef4b-4b68-af47-d550a11385d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:25.837755 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.830854 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-model-cache" pod "21c2b5d2-ef4b-4b68-af47-d550a11385d7" (UID: "21c2b5d2-ef4b-4b68-af47-d550a11385d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:25.837755 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.831438 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-model-cache" pod "49f28c58-bbe2-4110-8ac1-42e629c8be80" (UID: "49f28c58-bbe2-4110-8ac1-42e629c8be80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:25.840825 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.840774 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c2b5d2-ef4b-4b68-af47-d550a11385d7-kube-api-access-4lqgs" pod "21c2b5d2-ef4b-4b68-af47-d550a11385d7" (UID: "21c2b5d2-ef4b-4b68-af47-d550a11385d7"). InnerVolumeSpecName "kube-api-access-4lqgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:25.846850 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.846710 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-dshm" pod "49f28c58-bbe2-4110-8ac1-42e629c8be80" (UID: "49f28c58-bbe2-4110-8ac1-42e629c8be80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:25.846850 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.846721 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f28c58-bbe2-4110-8ac1-42e629c8be80-kube-api-access-7gv62" pod "49f28c58-bbe2-4110-8ac1-42e629c8be80" (UID: "49f28c58-bbe2-4110-8ac1-42e629c8be80"). InnerVolumeSpecName "kube-api-access-7gv62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:25.846850 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.846822 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49f28c58-bbe2-4110-8ac1-42e629c8be80-tls-certs" pod "49f28c58-bbe2-4110-8ac1-42e629c8be80" (UID: "49f28c58-bbe2-4110-8ac1-42e629c8be80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:25.847193 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.847173 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21c2b5d2-ef4b-4b68-af47-d550a11385d7-tls-certs" pod "21c2b5d2-ef4b-4b68-af47-d550a11385d7" (UID: "21c2b5d2-ef4b-4b68-af47-d550a11385d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:25.848452 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.848425 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-dshm" pod "21c2b5d2-ef4b-4b68-af47-d550a11385d7" (UID: "21c2b5d2-ef4b-4b68-af47-d550a11385d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:25.932004 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.931852 6006 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-dshm\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:25.932004 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.931902 6006 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-home\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:25.932004 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.931916 6006 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-tmp-dir\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:25.934378 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.931930 6006 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-dshm\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:25.934378 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.932474 6006 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49f28c58-bbe2-4110-8ac1-42e629c8be80-tls-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:25.934378 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.932488 6006 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21c2b5d2-ef4b-4b68-af47-d550a11385d7-tls-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:25.934378 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.932506 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lqgs\" (UniqueName: \"kubernetes.io/projected/21c2b5d2-ef4b-4b68-af47-d550a11385d7-kube-api-access-4lqgs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:25.934378 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.932522 6006 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21c2b5d2-ef4b-4b68-af47-d550a11385d7-model-cache\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:25.934378 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.932539 6006 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49f28c58-bbe2-4110-8ac1-42e629c8be80-model-cache\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:25.934378 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:25.932553 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gv62\" (UniqueName: \"kubernetes.io/projected/49f28c58-bbe2-4110-8ac1-42e629c8be80-kube-api-access-7gv62\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:26.170894 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.170853 6006 generic.go:356] "Generic (PLEG): container finished" podID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerID="b271c93432210844c022fbe220ab3c64946294a1fbb5338ae2eb145ec5b07e9f" exitCode=0 Oct 30 22:31:26.171352 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.170907 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" Oct 30 22:31:26.171352 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.170938 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" event={"ID":"21c2b5d2-ef4b-4b68-af47-d550a11385d7","Type":"ContainerDied","Data":"b271c93432210844c022fbe220ab3c64946294a1fbb5338ae2eb145ec5b07e9f"} Oct 30 22:31:26.171352 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.170984 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g" event={"ID":"21c2b5d2-ef4b-4b68-af47-d550a11385d7","Type":"ContainerDied","Data":"6388b1a5a04868dc1e0cf6cd1a32edf7f5cd106e0b811d59d5209b83a78ced3b"} Oct 30 22:31:26.171352 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.171008 6006 scope.go:122] "RemoveContainer" containerID="b271c93432210844c022fbe220ab3c64946294a1fbb5338ae2eb145ec5b07e9f" Oct 30 22:31:26.172627 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.172604 6006 generic.go:356] "Generic (PLEG): container finished" podID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerID="4b183b869e530fb82a87326affa9db7c50a8780269d475c2c6f6375d9d43ff5d" exitCode=0 Oct 30 22:31:26.172714 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.172651 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" Oct 30 22:31:26.172714 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.172670 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" event={"ID":"4e5d742b-c020-45ff-bd02-76663b0bb04a","Type":"ContainerDied","Data":"4b183b869e530fb82a87326affa9db7c50a8780269d475c2c6f6375d9d43ff5d"} Oct 30 22:31:26.172793 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.172711 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt" event={"ID":"4e5d742b-c020-45ff-bd02-76663b0bb04a","Type":"ContainerDied","Data":"119ed64f7de1e69ffc1364843fc873429a4b803a04eca78e18e7ad22a55afbe4"} Oct 30 22:31:26.178553 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.178522 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3d021e-97bzr" Oct 30 22:31:26.178677 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.178553 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd3d021e-97bzr" event={"ID":"c8c456dc-1d23-456b-b272-1431f457b971","Type":"ContainerDied","Data":"101008875df81859a5630c6d977e8ab8c7d16dfc81cb55f85cf9820eb48bc8cb"} Oct 30 22:31:26.179005 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.178982 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="101008875df81859a5630c6d977e8ab8c7d16dfc81cb55f85cf9820eb48bc8cb" Oct 30 22:31:26.180390 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.180206 6006 generic.go:356] "Generic (PLEG): container finished" podID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerID="1c9a8e6cf55ecbd8b352c708dcb5dfb9ce34d089ea5963246c051ee537cba547" exitCode=0 Oct 30 22:31:26.180390 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.180297 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" Oct 30 22:31:26.180390 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.180293 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" event={"ID":"49f28c58-bbe2-4110-8ac1-42e629c8be80","Type":"ContainerDied","Data":"1c9a8e6cf55ecbd8b352c708dcb5dfb9ce34d089ea5963246c051ee537cba547"} Oct 30 22:31:26.180563 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.180410 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss" event={"ID":"49f28c58-bbe2-4110-8ac1-42e629c8be80","Type":"ContainerDied","Data":"eba0f8cb5747fa8d31cfa80d6fd3932c72490219d64359cbf5f1e768a7264509"} Oct 30 22:31:26.182202 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.182182 6006 scope.go:122] "RemoveContainer" containerID="b271c93432210844c022fbe220ab3c64946294a1fbb5338ae2eb145ec5b07e9f" Oct 30 22:31:26.182817 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:31:26.182637 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b271c93432210844c022fbe220ab3c64946294a1fbb5338ae2eb145ec5b07e9f\": container with ID starting with b271c93432210844c022fbe220ab3c64946294a1fbb5338ae2eb145ec5b07e9f not found: ID does not exist" containerID="b271c93432210844c022fbe220ab3c64946294a1fbb5338ae2eb145ec5b07e9f" Oct 30 22:31:26.182817 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.182697 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b271c93432210844c022fbe220ab3c64946294a1fbb5338ae2eb145ec5b07e9f"} err="failed to get container status \"b271c93432210844c022fbe220ab3c64946294a1fbb5338ae2eb145ec5b07e9f\": rpc error: code = NotFound desc = could not find container \"b271c93432210844c022fbe220ab3c64946294a1fbb5338ae2eb145ec5b07e9f\": container with ID starting with b271c93432210844c022fbe220ab3c64946294a1fbb5338ae2eb145ec5b07e9f not found: ID does not exist" Oct 30 22:31:26.182817 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.182734 6006 scope.go:122] "RemoveContainer" containerID="4b183b869e530fb82a87326affa9db7c50a8780269d475c2c6f6375d9d43ff5d" Oct 30 22:31:26.197145 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.197108 6006 scope.go:122] "RemoveContainer" containerID="4b183b869e530fb82a87326affa9db7c50a8780269d475c2c6f6375d9d43ff5d" Oct 30 22:31:26.197669 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:31:26.197600 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b183b869e530fb82a87326affa9db7c50a8780269d475c2c6f6375d9d43ff5d\": container with ID starting with 4b183b869e530fb82a87326affa9db7c50a8780269d475c2c6f6375d9d43ff5d not found: ID does not exist" containerID="4b183b869e530fb82a87326affa9db7c50a8780269d475c2c6f6375d9d43ff5d" Oct 30 22:31:26.197669 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.197631 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b183b869e530fb82a87326affa9db7c50a8780269d475c2c6f6375d9d43ff5d"} err="failed to get container status \"4b183b869e530fb82a87326affa9db7c50a8780269d475c2c6f6375d9d43ff5d\": rpc error: code = NotFound desc = could not find container \"4b183b869e530fb82a87326affa9db7c50a8780269d475c2c6f6375d9d43ff5d\": container with ID starting with 4b183b869e530fb82a87326affa9db7c50a8780269d475c2c6f6375d9d43ff5d not found: ID does not exist" Oct 30 22:31:26.197669 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.197658 6006 scope.go:122] "RemoveContainer" containerID="1c9a8e6cf55ecbd8b352c708dcb5dfb9ce34d089ea5963246c051ee537cba547" Oct 30 22:31:26.207927 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.207901 6006 scope.go:122] "RemoveContainer" containerID="1c9a8e6cf55ecbd8b352c708dcb5dfb9ce34d089ea5963246c051ee537cba547" Oct 30 22:31:26.208353 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:31:26.208323 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c9a8e6cf55ecbd8b352c708dcb5dfb9ce34d089ea5963246c051ee537cba547\": container with ID starting with 1c9a8e6cf55ecbd8b352c708dcb5dfb9ce34d089ea5963246c051ee537cba547 not found: ID does not exist" containerID="1c9a8e6cf55ecbd8b352c708dcb5dfb9ce34d089ea5963246c051ee537cba547" Oct 30 22:31:26.208446 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.208363 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c9a8e6cf55ecbd8b352c708dcb5dfb9ce34d089ea5963246c051ee537cba547"} err="failed to get container status \"1c9a8e6cf55ecbd8b352c708dcb5dfb9ce34d089ea5963246c051ee537cba547\": rpc error: code = NotFound desc = could not find container \"1c9a8e6cf55ecbd8b352c708dcb5dfb9ce34d089ea5963246c051ee537cba547\": container with ID starting with 1c9a8e6cf55ecbd8b352c708dcb5dfb9ce34d089ea5963246c051ee537cba547 not found: ID does not exist" Oct 30 22:31:26.378610 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.378574 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g"] Oct 30 22:31:26.383584 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.383554 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-c9dbe6/premium-w5-c9dbe6-kserve-6bcb5cfb86-bgr6g"] Oct 30 22:31:26.489569 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.489488 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt"] Oct 30 22:31:26.493108 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.493080 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-c9dbe6/distinct-2-w5-c9dbe6-kserve-6c79d6678b-4hlkt"] Oct 30 22:31:26.601049 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.601012 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss"] Oct 30 22:31:26.609327 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.609285 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-c9dbe6/simulator-w5-c9dbe6-kserve-ffb8776b7-g8bss"] Oct 30 22:31:26.787195 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:26.787109 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:31:27.603865 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:27.603822 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" path="/var/lib/kubelet/pods/21c2b5d2-ef4b-4b68-af47-d550a11385d7/volumes" Oct 30 22:31:27.604546 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:27.604521 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" path="/var/lib/kubelet/pods/49f28c58-bbe2-4110-8ac1-42e629c8be80/volumes" Oct 30 22:31:27.604994 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:27.604973 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" path="/var/lib/kubelet/pods/4e5d742b-c020-45ff-bd02-76663b0bb04a/volumes" Oct 30 22:31:27.911129 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:31:27.911041 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:31:43.247894 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:43.247851 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5686cb-gngr8"] Oct 30 22:31:43.292046 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:43.292007 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5686cb-gngr8"] Oct 30 22:31:43.606507 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:43.606460 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9" path="/var/lib/kubelet/pods/7dab58ad-1dd5-46ba-beb2-b32ea5f7fce9/volumes" Oct 30 22:31:52.925319 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:52.925279 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt"] Oct 30 22:31:52.926059 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:52.926036 6006 cpu_manager.go:414] "containerMap: removing container" podUID="c8c456dc-1d23-456b-b272-1431f457b971" containerName="revoke-keys" Oct 30 22:31:52.926059 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:52.926059 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8c456dc-1d23-456b-b272-1431f457b971" containerName="revoke-keys" Oct 30 22:31:52.926274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:52.926075 6006 cpu_manager.go:414] "containerMap: removing container" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerName="main" Oct 30 22:31:52.926274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:52.926083 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerName="main" Oct 30 22:31:52.926274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:52.926104 6006 cpu_manager.go:414] "containerMap: removing container" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" Oct 30 22:31:52.926274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:52.926112 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" Oct 30 22:31:52.926274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:52.926140 6006 cpu_manager.go:414] "containerMap: removing container" podUID="260a3f69-701a-4ff2-a2cd-769957f80522" containerName="revoke-keys" Oct 30 22:31:52.926274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:52.926147 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="260a3f69-701a-4ff2-a2cd-769957f80522" containerName="revoke-keys" Oct 30 22:31:52.926274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:52.926162 6006 cpu_manager.go:414] "containerMap: removing container" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" Oct 30 22:31:52.926274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:52.926168 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" Oct 30 22:31:52.926701 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:52.926321 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8c456dc-1d23-456b-b272-1431f457b971" containerName="revoke-keys" Oct 30 22:31:52.926701 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:52.926338 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e5d742b-c020-45ff-bd02-76663b0bb04a" containerName="main" Oct 30 22:31:52.926701 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:52.926351 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="49f28c58-bbe2-4110-8ac1-42e629c8be80" containerName="main" Oct 30 22:31:52.926701 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:52.926366 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="260a3f69-701a-4ff2-a2cd-769957f80522" containerName="revoke-keys" Oct 30 22:31:52.926701 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:52.926376 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="21c2b5d2-ef4b-4b68-af47-d550a11385d7" containerName="main" Oct 30 22:31:52.930151 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:52.930125 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt" Oct 30 22:31:52.995458 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:52.995422 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbcf9f55-750b-4081-8c9c-636feed707e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt\" (UID: \"dbcf9f55-750b-4081-8c9c-636feed707e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt" Oct 30 22:31:52.995654 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:52.995482 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbcf9f55-750b-4081-8c9c-636feed707e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt\" (UID: \"dbcf9f55-750b-4081-8c9c-636feed707e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt" Oct 30 22:31:52.995654 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:52.995552 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p5xt\" (UniqueName: \"kubernetes.io/projected/dbcf9f55-750b-4081-8c9c-636feed707e6-kube-api-access-7p5xt\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt\" (UID: \"dbcf9f55-750b-4081-8c9c-636feed707e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt" Oct 30 22:31:53.030749 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:53.030716 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:31:53.030749 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:53.030744 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:31:53.035314 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:53.035278 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt"] Oct 30 22:31:53.096410 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:53.096371 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbcf9f55-750b-4081-8c9c-636feed707e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt\" (UID: \"dbcf9f55-750b-4081-8c9c-636feed707e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt" Oct 30 22:31:53.096620 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:53.096454 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p5xt\" (UniqueName: \"kubernetes.io/projected/dbcf9f55-750b-4081-8c9c-636feed707e6-kube-api-access-7p5xt\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt\" (UID: \"dbcf9f55-750b-4081-8c9c-636feed707e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt" Oct 30 22:31:53.096620 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:53.096589 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbcf9f55-750b-4081-8c9c-636feed707e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt\" (UID: \"dbcf9f55-750b-4081-8c9c-636feed707e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt" Oct 30 22:31:53.097205 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:53.097174 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbcf9f55-750b-4081-8c9c-636feed707e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt\" (UID: \"dbcf9f55-750b-4081-8c9c-636feed707e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt" Oct 30 22:31:53.098987 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:53.098952 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbcf9f55-750b-4081-8c9c-636feed707e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt\" (UID: \"dbcf9f55-750b-4081-8c9c-636feed707e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt" Oct 30 22:31:53.155889 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:53.155849 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p5xt\" (UniqueName: \"kubernetes.io/projected/dbcf9f55-750b-4081-8c9c-636feed707e6-kube-api-access-7p5xt\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt\" (UID: \"dbcf9f55-750b-4081-8c9c-636feed707e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt" Oct 30 22:31:53.247521 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:53.247433 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt" Oct 30 22:31:53.621786 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:53.621760 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt"] Oct 30 22:31:53.624867 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:31:53.624831 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbcf9f55_750b_4081_8c9c_636feed707e6.slice/crio-6a4b8abfa866aef50732cae62ef97ad4124203f4cc3942909fc3b9a3db199e1e WatchSource:0}: Error finding container 6a4b8abfa866aef50732cae62ef97ad4124203f4cc3942909fc3b9a3db199e1e: Status 404 returned error can't find the container with id 6a4b8abfa866aef50732cae62ef97ad4124203f4cc3942909fc3b9a3db199e1e Oct 30 22:31:54.395036 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:54.394997 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt" event={"ID":"dbcf9f55-750b-4081-8c9c-636feed707e6","Type":"ContainerStarted","Data":"ca106e78a3ecf3e0f5b77950388ae40d61151e203a5eb2e68a55e7673b49bebf"} Oct 30 22:31:54.395467 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:54.395060 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt" event={"ID":"dbcf9f55-750b-4081-8c9c-636feed707e6","Type":"ContainerStarted","Data":"6a4b8abfa866aef50732cae62ef97ad4124203f4cc3942909fc3b9a3db199e1e"} Oct 30 22:31:54.507663 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:54.507596 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt" podStartSLOduration=2.5075731230000002 podStartE2EDuration="2.507573123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:54.504693926 +0000 UTC m=+2517.056870229" watchObservedRunningTime="2026-10-30 22:31:54.507573123 +0000 UTC m=+2517.059749425" Oct 30 22:31:55.407801 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:55.407093 6006 generic.go:356] "Generic (PLEG): container finished" podID="dbcf9f55-750b-4081-8c9c-636feed707e6" containerID="ca106e78a3ecf3e0f5b77950388ae40d61151e203a5eb2e68a55e7673b49bebf" exitCode=0 Oct 30 22:31:55.407801 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:55.407195 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt" event={"ID":"dbcf9f55-750b-4081-8c9c-636feed707e6","Type":"ContainerDied","Data":"ca106e78a3ecf3e0f5b77950388ae40d61151e203a5eb2e68a55e7673b49bebf"} Oct 30 22:31:56.574002 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:56.573970 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt" Oct 30 22:31:56.607085 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:56.607045 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbcf9f55-750b-4081-8c9c-636feed707e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbcf9f55-750b-4081-8c9c-636feed707e6-maas-api-xks-ca\") pod \"dbcf9f55-750b-4081-8c9c-636feed707e6\" (UID: \"dbcf9f55-750b-4081-8c9c-636feed707e6\") " Oct 30 22:31:56.607280 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:56.607185 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbcf9f55-750b-4081-8c9c-636feed707e6-kube-api-access-7p5xt\" (UniqueName: \"kubernetes.io/projected/dbcf9f55-750b-4081-8c9c-636feed707e6-kube-api-access-7p5xt\") pod \"dbcf9f55-750b-4081-8c9c-636feed707e6\" (UID: \"dbcf9f55-750b-4081-8c9c-636feed707e6\") " Oct 30 22:31:56.607280 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:56.607235 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbcf9f55-750b-4081-8c9c-636feed707e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbcf9f55-750b-4081-8c9c-636feed707e6-maas-api-openshift-ca\") pod \"dbcf9f55-750b-4081-8c9c-636feed707e6\" (UID: \"dbcf9f55-750b-4081-8c9c-636feed707e6\") " Oct 30 22:31:56.607687 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:56.607653 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbcf9f55-750b-4081-8c9c-636feed707e6-maas-api-openshift-ca" pod "dbcf9f55-750b-4081-8c9c-636feed707e6" (UID: "dbcf9f55-750b-4081-8c9c-636feed707e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:56.607798 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:56.607754 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbcf9f55-750b-4081-8c9c-636feed707e6-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.609536 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:56.609509 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbcf9f55-750b-4081-8c9c-636feed707e6-maas-api-xks-ca" pod "dbcf9f55-750b-4081-8c9c-636feed707e6" (UID: "dbcf9f55-750b-4081-8c9c-636feed707e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:56.609645 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:56.609587 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbcf9f55-750b-4081-8c9c-636feed707e6-kube-api-access-7p5xt" pod "dbcf9f55-750b-4081-8c9c-636feed707e6" (UID: "dbcf9f55-750b-4081-8c9c-636feed707e6"). InnerVolumeSpecName "kube-api-access-7p5xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:56.712389 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:56.712293 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbcf9f55-750b-4081-8c9c-636feed707e6-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.712389 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:56.712336 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p5xt\" (UniqueName: \"kubernetes.io/projected/dbcf9f55-750b-4081-8c9c-636feed707e6-kube-api-access-7p5xt\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:57.422606 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:57.422563 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt" Oct 30 22:31:57.422806 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:57.422606 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt" event={"ID":"dbcf9f55-750b-4081-8c9c-636feed707e6","Type":"ContainerDied","Data":"6a4b8abfa866aef50732cae62ef97ad4124203f4cc3942909fc3b9a3db199e1e"} Oct 30 22:31:57.422806 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:31:57.422665 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a4b8abfa866aef50732cae62ef97ad4124203f4cc3942909fc3b9a3db199e1e" Oct 30 22:31:57.914740 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:31:57.914703 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:32:10.631629 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.631533 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx"] Oct 30 22:32:10.632121 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.632082 6006 cpu_manager.go:414] "containerMap: removing container" podUID="dbcf9f55-750b-4081-8c9c-636feed707e6" containerName="revoke-keys" Oct 30 22:32:10.632121 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.632105 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbcf9f55-750b-4081-8c9c-636feed707e6" containerName="revoke-keys" Oct 30 22:32:10.632239 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.632213 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbcf9f55-750b-4081-8c9c-636feed707e6" containerName="revoke-keys" Oct 30 22:32:10.637297 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.637270 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.687172 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.687140 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-c9dbe6-openshift-default-dockercfg-bpxxl\"" Oct 30 22:32:10.737783 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.737735 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx"] Oct 30 22:32:10.786634 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.786601 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-data\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.786813 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.786644 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-token\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.786813 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.786744 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-workload-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.786885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.786847 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-workload-certs\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.786930 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.786886 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb8f7\" (UniqueName: \"kubernetes.io/projected/4e02e796-9f01-40d1-9ccb-1701724d589b-kube-api-access-qb8f7\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.786969 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.786946 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-envoy\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.787003 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.786975 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-credential-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.787003 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.786997 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-podinfo\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.787071 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.787020 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e02e796-9f01-40d1-9ccb-1701724d589b-istiod-ca-cert\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.888343 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.888230 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-data\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.888343 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.888287 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-token\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.888343 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.888339 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-workload-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.888599 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.888497 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-workload-certs\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.888599 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.888578 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb8f7\" (UniqueName: \"kubernetes.io/projected/4e02e796-9f01-40d1-9ccb-1701724d589b-kube-api-access-qb8f7\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.888705 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.888629 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-envoy\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.888705 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.888677 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-credential-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.888812 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.888710 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-data\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.888812 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.888714 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-podinfo\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.888812 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.888763 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e02e796-9f01-40d1-9ccb-1701724d589b-istiod-ca-cert\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.888812 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.888770 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-workload-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.889017 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.888848 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-workload-certs\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.889196 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.889170 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-credential-socket\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.889422 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.889404 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e02e796-9f01-40d1-9ccb-1701724d589b-istiod-ca-cert\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.890988 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.890965 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-envoy\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.891364 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.891342 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-podinfo\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.947331 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.947289 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-token\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.947472 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.947422 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb8f7\" (UniqueName: \"kubernetes.io/projected/4e02e796-9f01-40d1-9ccb-1701724d589b-kube-api-access-qb8f7\") pod \"e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:10.964208 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:10.964169 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:11.246112 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:11.246027 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:32:11.399158 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:11.399125 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx"] Oct 30 22:32:11.399403 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:32:11.399378 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e02e796_9f01_40d1_9ccb_1701724d589b.slice/crio-42f8f4b4e62dd98f1ccf0c6ebc82a696bd5c0b8ce67c626e4a4232cd0f6a41b6 WatchSource:0}: Error finding container 42f8f4b4e62dd98f1ccf0c6ebc82a696bd5c0b8ce67c626e4a4232cd0f6a41b6: Status 404 returned error can't find the container with id 42f8f4b4e62dd98f1ccf0c6ebc82a696bd5c0b8ce67c626e4a4232cd0f6a41b6 Oct 30 22:32:11.402083 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:11.402052 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:32:11.402151 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:11.402116 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:32:11.402151 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:11.402146 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:32:11.526015 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:11.525909 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" event={"ID":"4e02e796-9f01-40d1-9ccb-1701724d589b","Type":"ContainerStarted","Data":"adfb75e09a86a233d9096720df20dff1d9a194d01034ad10ad012ea8455e45ad"} Oct 30 22:32:11.526015 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:11.525977 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" event={"ID":"4e02e796-9f01-40d1-9ccb-1701724d589b","Type":"ContainerStarted","Data":"42f8f4b4e62dd98f1ccf0c6ebc82a696bd5c0b8ce67c626e4a4232cd0f6a41b6"} Oct 30 22:32:11.646979 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:11.646910 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" podStartSLOduration=1.646891195 podStartE2EDuration="1.646891195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:11.642977098 +0000 UTC m=+2534.195153397" watchObservedRunningTime="2026-10-30 22:32:11.646891195 +0000 UTC m=+2534.199067495" Oct 30 22:32:11.964556 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:11.964517 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:12.966758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:12.966722 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 22:32:12.967142 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:12.966778 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" podUID="4e02e796-9f01-40d1-9ccb-1701724d589b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 22:32:13.965432 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:13.965386 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 22:32:13.965655 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:13.965469 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" podUID="4e02e796-9f01-40d1-9ccb-1701724d589b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 22:32:14.965630 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:14.965589 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 22:32:14.966134 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:14.965654 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" podUID="4e02e796-9f01-40d1-9ccb-1701724d589b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 22:32:15.965521 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:15.965480 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 22:32:15.965710 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:15.965537 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" podUID="4e02e796-9f01-40d1-9ccb-1701724d589b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 22:32:16.323272 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:16.322398 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8"] Oct 30 22:32:16.323272 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:16.322648 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8" podUID="2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f" containerName="maas-api" containerID="cri-o://c7f2b6691b55b08111662b3ba960ca134218f4fb6ab91e0f26316a7f604f4b4c" gracePeriod=30 Oct 30 22:32:16.700323 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:16.694385 6006 generic.go:356] "Generic (PLEG): container finished" podID="2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f" containerID="c7f2b6691b55b08111662b3ba960ca134218f4fb6ab91e0f26316a7f604f4b4c" exitCode=0 Oct 30 22:32:16.700323 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:16.694494 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8" event={"ID":"2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f","Type":"ContainerDied","Data":"c7f2b6691b55b08111662b3ba960ca134218f4fb6ab91e0f26316a7f604f4b4c"} Oct 30 22:32:16.867296 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:16.863356 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8" Oct 30 22:32:16.892303 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:16.892268 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-metrics-certs\") pod \"2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f\" (UID: \"2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f\") " Oct 30 22:32:16.892485 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:16.892325 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-openshift-service-ca\") pod \"2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f\" (UID: \"2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f\") " Oct 30 22:32:16.892485 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:16.892388 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-maas-api-tls\") pod \"2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f\" (UID: \"2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f\") " Oct 30 22:32:16.892581 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:16.892509 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-kube-api-access-79t6n\" (UniqueName: \"kubernetes.io/projected/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-kube-api-access-79t6n\") pod \"2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f\" (UID: \"2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f\") " Oct 30 22:32:16.893485 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:16.893456 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-openshift-service-ca" pod "2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f" (UID: "2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:16.894995 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:16.894970 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-kube-api-access-79t6n" pod "2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f" (UID: "2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f"). InnerVolumeSpecName "kube-api-access-79t6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:16.895353 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:16.895337 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-metrics-certs" pod "2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f" (UID: "2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:16.895591 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:16.895565 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-maas-api-tls" pod "2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f" (UID: "2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:16.964735 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:16.964631 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 22:32:16.964735 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:16.964714 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" podUID="4e02e796-9f01-40d1-9ccb-1701724d589b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 22:32:16.994620 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:16.994580 6006 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-metrics-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:16.994620 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:16.994621 6006 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-openshift-service-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:16.995199 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:16.994637 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-maas-api-tls\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:16.995199 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:16.994650 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79t6n\" (UniqueName: \"kubernetes.io/projected/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f-kube-api-access-79t6n\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:17.703110 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:17.703076 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8" event={"ID":"2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f","Type":"ContainerDied","Data":"ab9975f82faac96ff5995164e88ca4ad0c4ef690f0b106e6cef481a84d4a5ec7"} Oct 30 22:32:17.703110 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:17.703108 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8" Oct 30 22:32:17.703436 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:17.703139 6006 scope.go:122] "RemoveContainer" containerID="c7f2b6691b55b08111662b3ba960ca134218f4fb6ab91e0f26316a7f604f4b4c" Oct 30 22:32:17.859923 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:17.859881 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8"] Oct 30 22:32:17.906585 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:17.906491 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c9dbe6-76f984bbc7-qzvr8"] Oct 30 22:32:17.965058 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:17.965015 6006 patch_prober.go:29] interesting pod/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 22:32:17.965280 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:17.965084 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" podUID="4e02e796-9f01-40d1-9ccb-1701724d589b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 22:32:18.387176 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:18.387136 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc"] Oct 30 22:32:18.391629 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:18.391596 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c9dbe6-2zkpc"] Oct 30 22:32:19.113285 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:19.113213 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:19.113793 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:19.113765 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:19.114956 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:19.114933 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:19.604647 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:19.604611 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15614cea-f680-44fb-bc46-7d632d5c6770" path="/var/lib/kubelet/pods/15614cea-f680-44fb-bc46-7d632d5c6770/volumes" Oct 30 22:32:19.605189 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:19.605169 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f" path="/var/lib/kubelet/pods/2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f/volumes" Oct 30 22:32:25.663112 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:25.663068 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx"] Oct 30 22:32:25.663698 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:25.663418 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" podUID="4e02e796-9f01-40d1-9ccb-1701724d589b" containerName="istio-proxy" containerID="cri-o://adfb75e09a86a233d9096720df20dff1d9a194d01034ad10ad012ea8455e45ad" gracePeriod=30 Oct 30 22:32:27.911564 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:32:27.911515 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:32:30.809031 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:30.808996 6006 generic.go:356] "Generic (PLEG): container finished" podID="4e02e796-9f01-40d1-9ccb-1701724d589b" containerID="adfb75e09a86a233d9096720df20dff1d9a194d01034ad10ad012ea8455e45ad" exitCode=0 Oct 30 22:32:30.809520 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:30.809070 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" event={"ID":"4e02e796-9f01-40d1-9ccb-1701724d589b","Type":"ContainerDied","Data":"adfb75e09a86a233d9096720df20dff1d9a194d01034ad10ad012ea8455e45ad"} Oct 30 22:32:30.965152 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:30.965124 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:31.101333 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.101294 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-token\" (UniqueName: \"kubernetes.io/projected/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-token\") pod \"4e02e796-9f01-40d1-9ccb-1701724d589b\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " Oct 30 22:32:31.101333 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.101335 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-workload-socket\") pod \"4e02e796-9f01-40d1-9ccb-1701724d589b\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " Oct 30 22:32:31.101619 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.101389 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-credential-socket\") pod \"4e02e796-9f01-40d1-9ccb-1701724d589b\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " Oct 30 22:32:31.101619 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.101452 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-envoy\") pod \"4e02e796-9f01-40d1-9ccb-1701724d589b\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " Oct 30 22:32:31.101619 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.101483 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e02e796-9f01-40d1-9ccb-1701724d589b-kube-api-access-qb8f7\" (UniqueName: \"kubernetes.io/projected/4e02e796-9f01-40d1-9ccb-1701724d589b-kube-api-access-qb8f7\") pod \"4e02e796-9f01-40d1-9ccb-1701724d589b\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " Oct 30 22:32:31.101619 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.101533 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-data\") pod \"4e02e796-9f01-40d1-9ccb-1701724d589b\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " Oct 30 22:32:31.101619 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.101560 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-podinfo\") pod \"4e02e796-9f01-40d1-9ccb-1701724d589b\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " Oct 30 22:32:31.101904 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.101641 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-workload-certs\") pod \"4e02e796-9f01-40d1-9ccb-1701724d589b\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " Oct 30 22:32:31.101904 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.101681 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e02e796-9f01-40d1-9ccb-1701724d589b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e02e796-9f01-40d1-9ccb-1701724d589b-istiod-ca-cert\") pod \"4e02e796-9f01-40d1-9ccb-1701724d589b\" (UID: \"4e02e796-9f01-40d1-9ccb-1701724d589b\") " Oct 30 22:32:31.101904 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.101743 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-workload-socket" pod "4e02e796-9f01-40d1-9ccb-1701724d589b" (UID: "4e02e796-9f01-40d1-9ccb-1701724d589b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:31.101904 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.101799 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-credential-socket" pod "4e02e796-9f01-40d1-9ccb-1701724d589b" (UID: "4e02e796-9f01-40d1-9ccb-1701724d589b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:31.102122 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.102004 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-workload-certs" pod "4e02e796-9f01-40d1-9ccb-1701724d589b" (UID: "4e02e796-9f01-40d1-9ccb-1701724d589b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:31.102122 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.102020 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-data" pod "4e02e796-9f01-40d1-9ccb-1701724d589b" (UID: "4e02e796-9f01-40d1-9ccb-1701724d589b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:31.102311 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.102289 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:31.102395 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.102317 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:31.102395 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.102331 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:31.102395 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.102345 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:31.102561 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.102392 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e02e796-9f01-40d1-9ccb-1701724d589b-istiod-ca-cert" pod "4e02e796-9f01-40d1-9ccb-1701724d589b" (UID: "4e02e796-9f01-40d1-9ccb-1701724d589b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:31.104122 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.104081 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-token" pod "4e02e796-9f01-40d1-9ccb-1701724d589b" (UID: "4e02e796-9f01-40d1-9ccb-1701724d589b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:31.104400 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.104376 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-podinfo" pod "4e02e796-9f01-40d1-9ccb-1701724d589b" (UID: "4e02e796-9f01-40d1-9ccb-1701724d589b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:32:31.104491 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.104451 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-envoy" pod "4e02e796-9f01-40d1-9ccb-1701724d589b" (UID: "4e02e796-9f01-40d1-9ccb-1701724d589b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:31.104679 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.104652 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e02e796-9f01-40d1-9ccb-1701724d589b-kube-api-access-qb8f7" pod "4e02e796-9f01-40d1-9ccb-1701724d589b" (UID: "4e02e796-9f01-40d1-9ccb-1701724d589b"). InnerVolumeSpecName "kube-api-access-qb8f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:31.204014 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.203923 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e02e796-9f01-40d1-9ccb-1701724d589b-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:31.204014 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.203956 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:31.204014 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.203966 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:31.204014 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.203977 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb8f7\" (UniqueName: \"kubernetes.io/projected/4e02e796-9f01-40d1-9ccb-1701724d589b-kube-api-access-qb8f7\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:31.204014 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.203987 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e02e796-9f01-40d1-9ccb-1701724d589b-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:31.818455 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.818424 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" Oct 30 22:32:31.818943 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.818423 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx" event={"ID":"4e02e796-9f01-40d1-9ccb-1701724d589b","Type":"ContainerDied","Data":"42f8f4b4e62dd98f1ccf0c6ebc82a696bd5c0b8ce67c626e4a4232cd0f6a41b6"} Oct 30 22:32:31.818943 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.818571 6006 scope.go:122] "RemoveContainer" containerID="adfb75e09a86a233d9096720df20dff1d9a194d01034ad10ad012ea8455e45ad" Oct 30 22:32:31.978025 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:31.977936 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx"] Oct 30 22:32:32.024276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:32.024207 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c9dbe6-openshift-default-59bd48d5d5-6skvx"] Oct 30 22:32:32.885020 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:32.884978 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p"] Oct 30 22:32:32.885549 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:32.885534 6006 cpu_manager.go:414] "containerMap: removing container" podUID="2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f" containerName="maas-api" Oct 30 22:32:32.885609 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:32.885550 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f" containerName="maas-api" Oct 30 22:32:32.885609 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:32.885566 6006 cpu_manager.go:414] "containerMap: removing container" podUID="4e02e796-9f01-40d1-9ccb-1701724d589b" containerName="istio-proxy" Oct 30 22:32:32.885609 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:32.885571 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e02e796-9f01-40d1-9ccb-1701724d589b" containerName="istio-proxy" Oct 30 22:32:32.885761 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:32.885668 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bfa3a54-fb74-4985-85c2-df5d7fb3ed1f" containerName="maas-api" Oct 30 22:32:32.885761 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:32.885680 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e02e796-9f01-40d1-9ccb-1701724d589b" containerName="istio-proxy" Oct 30 22:32:32.890354 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:32.890326 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p" Oct 30 22:32:32.945411 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:32.945379 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:32:32.948355 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:32.948316 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p"] Oct 30 22:32:32.990714 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:32.990668 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:32:32.991820 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:32.991785 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:32:32.991943 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:32.991818 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:32:33.033762 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:33.033721 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9083308-566b-4535-b9f0-630f6775bce0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p\" (UID: \"e9083308-566b-4535-b9f0-630f6775bce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p" Oct 30 22:32:33.033941 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:33.033776 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9083308-566b-4535-b9f0-630f6775bce0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p\" (UID: \"e9083308-566b-4535-b9f0-630f6775bce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p" Oct 30 22:32:33.033941 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:33.033909 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78jft\" (UniqueName: \"kubernetes.io/projected/e9083308-566b-4535-b9f0-630f6775bce0-kube-api-access-78jft\") pod \"maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p\" (UID: \"e9083308-566b-4535-b9f0-630f6775bce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p" Oct 30 22:32:33.134941 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:33.134906 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9083308-566b-4535-b9f0-630f6775bce0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p\" (UID: \"e9083308-566b-4535-b9f0-630f6775bce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p" Oct 30 22:32:33.135170 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:33.134961 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9083308-566b-4535-b9f0-630f6775bce0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p\" (UID: \"e9083308-566b-4535-b9f0-630f6775bce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p" Oct 30 22:32:33.135170 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:33.135046 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78jft\" (UniqueName: \"kubernetes.io/projected/e9083308-566b-4535-b9f0-630f6775bce0-kube-api-access-78jft\") pod \"maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p\" (UID: \"e9083308-566b-4535-b9f0-630f6775bce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p" Oct 30 22:32:33.135756 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:33.135734 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9083308-566b-4535-b9f0-630f6775bce0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p\" (UID: \"e9083308-566b-4535-b9f0-630f6775bce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p" Oct 30 22:32:33.137385 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:33.137363 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9083308-566b-4535-b9f0-630f6775bce0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p\" (UID: \"e9083308-566b-4535-b9f0-630f6775bce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p" Oct 30 22:32:33.196332 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:33.196298 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78jft\" (UniqueName: \"kubernetes.io/projected/e9083308-566b-4535-b9f0-630f6775bce0-kube-api-access-78jft\") pod \"maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p\" (UID: \"e9083308-566b-4535-b9f0-630f6775bce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p" Oct 30 22:32:33.204157 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:33.204128 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p" Oct 30 22:32:33.606764 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:33.606726 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e02e796-9f01-40d1-9ccb-1701724d589b" path="/var/lib/kubelet/pods/4e02e796-9f01-40d1-9ccb-1701724d589b/volumes" Oct 30 22:32:33.616974 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:33.616946 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p"] Oct 30 22:32:33.619133 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:32:33.619095 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9083308_566b_4535_b9f0_630f6775bce0.slice/crio-d0aefa5e4f1434bfc75933dfa4bd638251470cc10ccdd7f6a850ce789c21c438 WatchSource:0}: Error finding container d0aefa5e4f1434bfc75933dfa4bd638251470cc10ccdd7f6a850ce789c21c438: Status 404 returned error can't find the container with id d0aefa5e4f1434bfc75933dfa4bd638251470cc10ccdd7f6a850ce789c21c438 Oct 30 22:32:33.836799 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:33.836766 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p" event={"ID":"e9083308-566b-4535-b9f0-630f6775bce0","Type":"ContainerStarted","Data":"d0aefa5e4f1434bfc75933dfa4bd638251470cc10ccdd7f6a850ce789c21c438"} Oct 30 22:32:34.846224 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:34.846186 6006 generic.go:356] "Generic (PLEG): container finished" podID="e9083308-566b-4535-b9f0-630f6775bce0" containerID="5c57e4e8197fc693d6b6de3d29c1f2fd419afac4ee4b296ff63160d46e5e6149" exitCode=0 Oct 30 22:32:34.846701 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:34.846283 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p" event={"ID":"e9083308-566b-4535-b9f0-630f6775bce0","Type":"ContainerDied","Data":"5c57e4e8197fc693d6b6de3d29c1f2fd419afac4ee4b296ff63160d46e5e6149"} Oct 30 22:32:36.018158 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:36.018126 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p" Oct 30 22:32:36.171414 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:36.171323 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9083308-566b-4535-b9f0-630f6775bce0-kube-api-access-78jft\" (UniqueName: \"kubernetes.io/projected/e9083308-566b-4535-b9f0-630f6775bce0-kube-api-access-78jft\") pod \"e9083308-566b-4535-b9f0-630f6775bce0\" (UID: \"e9083308-566b-4535-b9f0-630f6775bce0\") " Oct 30 22:32:36.171597 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:36.171508 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9083308-566b-4535-b9f0-630f6775bce0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9083308-566b-4535-b9f0-630f6775bce0-maas-api-xks-ca\") pod \"e9083308-566b-4535-b9f0-630f6775bce0\" (UID: \"e9083308-566b-4535-b9f0-630f6775bce0\") " Oct 30 22:32:36.171597 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:36.171563 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9083308-566b-4535-b9f0-630f6775bce0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9083308-566b-4535-b9f0-630f6775bce0-maas-api-openshift-ca\") pod \"e9083308-566b-4535-b9f0-630f6775bce0\" (UID: \"e9083308-566b-4535-b9f0-630f6775bce0\") " Oct 30 22:32:36.171955 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:36.171930 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9083308-566b-4535-b9f0-630f6775bce0-maas-api-openshift-ca" pod "e9083308-566b-4535-b9f0-630f6775bce0" (UID: "e9083308-566b-4535-b9f0-630f6775bce0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:36.173672 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:36.173650 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9083308-566b-4535-b9f0-630f6775bce0-maas-api-xks-ca" pod "e9083308-566b-4535-b9f0-630f6775bce0" (UID: "e9083308-566b-4535-b9f0-630f6775bce0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:36.173672 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:36.173659 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9083308-566b-4535-b9f0-630f6775bce0-kube-api-access-78jft" pod "e9083308-566b-4535-b9f0-630f6775bce0" (UID: "e9083308-566b-4535-b9f0-630f6775bce0"). InnerVolumeSpecName "kube-api-access-78jft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:36.272828 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:36.272787 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78jft\" (UniqueName: \"kubernetes.io/projected/e9083308-566b-4535-b9f0-630f6775bce0-kube-api-access-78jft\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:36.272828 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:36.272823 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9083308-566b-4535-b9f0-630f6775bce0-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:36.272828 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:36.272833 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9083308-566b-4535-b9f0-630f6775bce0-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:36.863321 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:36.863291 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p" Oct 30 22:32:36.863498 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:36.863283 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p" event={"ID":"e9083308-566b-4535-b9f0-630f6775bce0","Type":"ContainerDied","Data":"d0aefa5e4f1434bfc75933dfa4bd638251470cc10ccdd7f6a850ce789c21c438"} Oct 30 22:32:36.863498 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:36.863419 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0aefa5e4f1434bfc75933dfa4bd638251470cc10ccdd7f6a850ce789c21c438" Oct 30 22:32:37.283747 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:37.283645 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn"] Oct 30 22:32:37.305514 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:37.305469 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk"] Oct 30 22:32:37.329727 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:37.329687 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569"] Oct 30 22:32:42.370104 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:42.370069 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:32:52.307489 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.307424 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" containerID="cri-o://947682e4a57dd01bffdc20aab0a087343fca49ca45f8637b1064679504cc061e" gracePeriod=45 Oct 30 22:32:52.328293 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.328238 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" containerID="cri-o://98806554a9c31588ac1844508be03fb6345e9d1f18a47b8492d7f12e5d665478" gracePeriod=45 Oct 30 22:32:52.352280 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.352225 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" containerID="cri-o://a6ed2634bf8a97594ab7a9ad2bf794a439ba2787985c34091decf53214b42034" gracePeriod=45 Oct 30 22:32:52.658058 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.658026 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:32:52.688904 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.688877 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:32:52.693019 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.692998 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:32:52.768593 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.768555 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9fc3151-cd09-4454-89c0-dc57204054ca-tls-certs\" (UniqueName: \"kubernetes.io/secret/a9fc3151-cd09-4454-89c0-dc57204054ca-tls-certs\") pod \"a9fc3151-cd09-4454-89c0-dc57204054ca\" (UID: \"a9fc3151-cd09-4454-89c0-dc57204054ca\") " Oct 30 22:32:52.768792 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.768616 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe28bccf-50cc-4c80-b29d-47810d3aabd2-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe28bccf-50cc-4c80-b29d-47810d3aabd2-tls-certs\") pod \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\" (UID: \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\") " Oct 30 22:32:52.768792 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.768667 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-model-cache\") pod \"a9fc3151-cd09-4454-89c0-dc57204054ca\" (UID: \"a9fc3151-cd09-4454-89c0-dc57204054ca\") " Oct 30 22:32:52.768792 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.768746 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-tmp-dir\") pod \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\" (UID: \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\") " Oct 30 22:32:52.768964 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.768812 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-home\" (UniqueName: \"kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-home\") pod \"08f28fb4-f013-4385-991d-742c9e38f4b2\" (UID: \"08f28fb4-f013-4385-991d-742c9e38f4b2\") " Oct 30 22:32:52.769024 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.768972 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-model-cache" pod "a9fc3151-cd09-4454-89c0-dc57204054ca" (UID: "a9fc3151-cd09-4454-89c0-dc57204054ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:52.769317 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.769183 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-tmp-dir" pod "fe28bccf-50cc-4c80-b29d-47810d3aabd2" (UID: "fe28bccf-50cc-4c80-b29d-47810d3aabd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:52.769317 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.769297 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-home" pod "08f28fb4-f013-4385-991d-742c9e38f4b2" (UID: "08f28fb4-f013-4385-991d-742c9e38f4b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:52.769548 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.769512 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-home\" (UniqueName: \"kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-home\") pod \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\" (UID: \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\") " Oct 30 22:32:52.770105 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.769552 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-tmp-dir\") pod \"a9fc3151-cd09-4454-89c0-dc57204054ca\" (UID: \"a9fc3151-cd09-4454-89c0-dc57204054ca\") " Oct 30 22:32:52.770105 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.769584 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-home\" (UniqueName: \"kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-home\") pod \"a9fc3151-cd09-4454-89c0-dc57204054ca\" (UID: \"a9fc3151-cd09-4454-89c0-dc57204054ca\") " Oct 30 22:32:52.770105 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.769611 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-dshm\") pod \"a9fc3151-cd09-4454-89c0-dc57204054ca\" (UID: \"a9fc3151-cd09-4454-89c0-dc57204054ca\") " Oct 30 22:32:52.770105 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.769649 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe28bccf-50cc-4c80-b29d-47810d3aabd2-kube-api-access-bm2m5\" (UniqueName: \"kubernetes.io/projected/fe28bccf-50cc-4c80-b29d-47810d3aabd2-kube-api-access-bm2m5\") pod \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\" (UID: \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\") " Oct 30 22:32:52.770105 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.769678 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08f28fb4-f013-4385-991d-742c9e38f4b2-kube-api-access-h7m2r\" (UniqueName: \"kubernetes.io/projected/08f28fb4-f013-4385-991d-742c9e38f4b2-kube-api-access-h7m2r\") pod \"08f28fb4-f013-4385-991d-742c9e38f4b2\" (UID: \"08f28fb4-f013-4385-991d-742c9e38f4b2\") " Oct 30 22:32:52.770105 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.769753 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-tmp-dir\") pod \"08f28fb4-f013-4385-991d-742c9e38f4b2\" (UID: \"08f28fb4-f013-4385-991d-742c9e38f4b2\") " Oct 30 22:32:52.770105 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.769788 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-model-cache\") pod \"08f28fb4-f013-4385-991d-742c9e38f4b2\" (UID: \"08f28fb4-f013-4385-991d-742c9e38f4b2\") " Oct 30 22:32:52.770105 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.769804 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-home" pod "fe28bccf-50cc-4c80-b29d-47810d3aabd2" (UID: "fe28bccf-50cc-4c80-b29d-47810d3aabd2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:52.770105 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.769840 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08f28fb4-f013-4385-991d-742c9e38f4b2-tls-certs\" (UniqueName: \"kubernetes.io/secret/08f28fb4-f013-4385-991d-742c9e38f4b2-tls-certs\") pod \"08f28fb4-f013-4385-991d-742c9e38f4b2\" (UID: \"08f28fb4-f013-4385-991d-742c9e38f4b2\") " Oct 30 22:32:52.770105 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.769921 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-dshm\") pod \"08f28fb4-f013-4385-991d-742c9e38f4b2\" (UID: \"08f28fb4-f013-4385-991d-742c9e38f4b2\") " Oct 30 22:32:52.770105 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.769978 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-model-cache\") pod \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\" (UID: \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\") " Oct 30 22:32:52.770105 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.770035 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-dshm\") pod \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\" (UID: \"fe28bccf-50cc-4c80-b29d-47810d3aabd2\") " Oct 30 22:32:52.770105 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.770066 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-model-cache" pod "08f28fb4-f013-4385-991d-742c9e38f4b2" (UID: "08f28fb4-f013-4385-991d-742c9e38f4b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:52.770105 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.770083 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9fc3151-cd09-4454-89c0-dc57204054ca-kube-api-access-4db9n\" (UniqueName: \"kubernetes.io/projected/a9fc3151-cd09-4454-89c0-dc57204054ca-kube-api-access-4db9n\") pod \"a9fc3151-cd09-4454-89c0-dc57204054ca\" (UID: \"a9fc3151-cd09-4454-89c0-dc57204054ca\") " Oct 30 22:32:52.770943 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.770266 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-tmp-dir" pod "a9fc3151-cd09-4454-89c0-dc57204054ca" (UID: "a9fc3151-cd09-4454-89c0-dc57204054ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:52.770943 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.770285 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-tmp-dir" pod "08f28fb4-f013-4385-991d-742c9e38f4b2" (UID: "08f28fb4-f013-4385-991d-742c9e38f4b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:52.770943 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.770308 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-home" pod "a9fc3151-cd09-4454-89c0-dc57204054ca" (UID: "a9fc3151-cd09-4454-89c0-dc57204054ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:52.770943 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.770511 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-model-cache" pod "fe28bccf-50cc-4c80-b29d-47810d3aabd2" (UID: "fe28bccf-50cc-4c80-b29d-47810d3aabd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:52.770943 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.770680 6006 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-model-cache\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:52.770943 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.770700 6006 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-model-cache\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:52.770943 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.770715 6006 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-tmp-dir\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:52.770943 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.770729 6006 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-home\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:52.770943 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.770742 6006 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-home\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:52.770943 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.770755 6006 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-tmp-dir\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:52.770943 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.770769 6006 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-home\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:52.770943 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.770782 6006 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-tmp-dir\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:52.770943 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.770796 6006 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-model-cache\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:52.774326 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.774298 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-dshm" pod "a9fc3151-cd09-4454-89c0-dc57204054ca" (UID: "a9fc3151-cd09-4454-89c0-dc57204054ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:52.774445 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.774425 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe28bccf-50cc-4c80-b29d-47810d3aabd2-tls-certs" pod "fe28bccf-50cc-4c80-b29d-47810d3aabd2" (UID: "fe28bccf-50cc-4c80-b29d-47810d3aabd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:52.774527 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.774442 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9fc3151-cd09-4454-89c0-dc57204054ca-kube-api-access-4db9n" pod "a9fc3151-cd09-4454-89c0-dc57204054ca" (UID: "a9fc3151-cd09-4454-89c0-dc57204054ca"). InnerVolumeSpecName "kube-api-access-4db9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:52.774527 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.774404 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe28bccf-50cc-4c80-b29d-47810d3aabd2-kube-api-access-bm2m5" pod "fe28bccf-50cc-4c80-b29d-47810d3aabd2" (UID: "fe28bccf-50cc-4c80-b29d-47810d3aabd2"). InnerVolumeSpecName "kube-api-access-bm2m5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:52.774527 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.774424 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f28fb4-f013-4385-991d-742c9e38f4b2-kube-api-access-h7m2r" pod "08f28fb4-f013-4385-991d-742c9e38f4b2" (UID: "08f28fb4-f013-4385-991d-742c9e38f4b2"). InnerVolumeSpecName "kube-api-access-h7m2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:52.774527 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.774326 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9fc3151-cd09-4454-89c0-dc57204054ca-tls-certs" pod "a9fc3151-cd09-4454-89c0-dc57204054ca" (UID: "a9fc3151-cd09-4454-89c0-dc57204054ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:52.776020 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.775990 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08f28fb4-f013-4385-991d-742c9e38f4b2-tls-certs" pod "08f28fb4-f013-4385-991d-742c9e38f4b2" (UID: "08f28fb4-f013-4385-991d-742c9e38f4b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:52.776129 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.776033 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-dshm" pod "08f28fb4-f013-4385-991d-742c9e38f4b2" (UID: "08f28fb4-f013-4385-991d-742c9e38f4b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:52.776129 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.776040 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-dshm" pod "fe28bccf-50cc-4c80-b29d-47810d3aabd2" (UID: "fe28bccf-50cc-4c80-b29d-47810d3aabd2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:52.871889 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.871805 6006 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9fc3151-cd09-4454-89c0-dc57204054ca-dshm\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:52.871889 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.871837 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm2m5\" (UniqueName: \"kubernetes.io/projected/fe28bccf-50cc-4c80-b29d-47810d3aabd2-kube-api-access-bm2m5\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:52.871889 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.871848 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7m2r\" (UniqueName: \"kubernetes.io/projected/08f28fb4-f013-4385-991d-742c9e38f4b2-kube-api-access-h7m2r\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:52.871889 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.871857 6006 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08f28fb4-f013-4385-991d-742c9e38f4b2-tls-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:52.871889 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.871866 6006 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08f28fb4-f013-4385-991d-742c9e38f4b2-dshm\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:52.871889 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.871874 6006 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe28bccf-50cc-4c80-b29d-47810d3aabd2-dshm\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:52.871889 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.871882 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4db9n\" (UniqueName: \"kubernetes.io/projected/a9fc3151-cd09-4454-89c0-dc57204054ca-kube-api-access-4db9n\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:52.871889 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.871893 6006 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9fc3151-cd09-4454-89c0-dc57204054ca-tls-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:52.872357 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.871903 6006 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe28bccf-50cc-4c80-b29d-47810d3aabd2-tls-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:52.983913 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.978834 6006 generic.go:356] "Generic (PLEG): container finished" podID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerID="98806554a9c31588ac1844508be03fb6345e9d1f18a47b8492d7f12e5d665478" exitCode=0 Oct 30 22:32:52.983913 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.978894 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" Oct 30 22:32:52.983913 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.978898 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" event={"ID":"a9fc3151-cd09-4454-89c0-dc57204054ca","Type":"ContainerDied","Data":"98806554a9c31588ac1844508be03fb6345e9d1f18a47b8492d7f12e5d665478"} Oct 30 22:32:52.983913 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.978983 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk" event={"ID":"a9fc3151-cd09-4454-89c0-dc57204054ca","Type":"ContainerDied","Data":"e59ac3e7705b80edfb1fe162afdae4be81250780345d62b364f1535de1d30d3a"} Oct 30 22:32:52.983913 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.979009 6006 scope.go:122] "RemoveContainer" containerID="98806554a9c31588ac1844508be03fb6345e9d1f18a47b8492d7f12e5d665478" Oct 30 22:32:52.983913 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.983027 6006 generic.go:356] "Generic (PLEG): container finished" podID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerID="a6ed2634bf8a97594ab7a9ad2bf794a439ba2787985c34091decf53214b42034" exitCode=0 Oct 30 22:32:52.983913 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.983177 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" event={"ID":"fe28bccf-50cc-4c80-b29d-47810d3aabd2","Type":"ContainerDied","Data":"a6ed2634bf8a97594ab7a9ad2bf794a439ba2787985c34091decf53214b42034"} Oct 30 22:32:52.983913 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.983207 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" event={"ID":"fe28bccf-50cc-4c80-b29d-47810d3aabd2","Type":"ContainerDied","Data":"a6473218f6a74d88b674fa71888399b55ae9ada188d24e8ceb121463fded629e"} Oct 30 22:32:52.983913 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.983306 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569" Oct 30 22:32:52.986425 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.986344 6006 generic.go:356] "Generic (PLEG): container finished" podID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerID="947682e4a57dd01bffdc20aab0a087343fca49ca45f8637b1064679504cc061e" exitCode=0 Oct 30 22:32:52.986576 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.986435 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" event={"ID":"08f28fb4-f013-4385-991d-742c9e38f4b2","Type":"ContainerDied","Data":"947682e4a57dd01bffdc20aab0a087343fca49ca45f8637b1064679504cc061e"} Oct 30 22:32:52.986576 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.986480 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" event={"ID":"08f28fb4-f013-4385-991d-742c9e38f4b2","Type":"ContainerDied","Data":"194db9587d5f1243135e553a2caae2b1b1e19ae7783a0678a2481ffd70294109"} Oct 30 22:32:52.986576 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.986514 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn" Oct 30 22:32:52.995628 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.995598 6006 scope.go:122] "RemoveContainer" containerID="98806554a9c31588ac1844508be03fb6345e9d1f18a47b8492d7f12e5d665478" Oct 30 22:32:52.995981 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:32:52.995953 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98806554a9c31588ac1844508be03fb6345e9d1f18a47b8492d7f12e5d665478\": container with ID starting with 98806554a9c31588ac1844508be03fb6345e9d1f18a47b8492d7f12e5d665478 not found: ID does not exist" containerID="98806554a9c31588ac1844508be03fb6345e9d1f18a47b8492d7f12e5d665478" Oct 30 22:32:52.996078 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.995990 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98806554a9c31588ac1844508be03fb6345e9d1f18a47b8492d7f12e5d665478"} err="failed to get container status \"98806554a9c31588ac1844508be03fb6345e9d1f18a47b8492d7f12e5d665478\": rpc error: code = NotFound desc = could not find container \"98806554a9c31588ac1844508be03fb6345e9d1f18a47b8492d7f12e5d665478\": container with ID starting with 98806554a9c31588ac1844508be03fb6345e9d1f18a47b8492d7f12e5d665478 not found: ID does not exist" Oct 30 22:32:52.996078 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:52.996015 6006 scope.go:122] "RemoveContainer" containerID="a6ed2634bf8a97594ab7a9ad2bf794a439ba2787985c34091decf53214b42034" Oct 30 22:32:53.027463 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:53.026710 6006 scope.go:122] "RemoveContainer" containerID="a6ed2634bf8a97594ab7a9ad2bf794a439ba2787985c34091decf53214b42034" Oct 30 22:32:53.027463 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:32:53.027106 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6ed2634bf8a97594ab7a9ad2bf794a439ba2787985c34091decf53214b42034\": container with ID starting with a6ed2634bf8a97594ab7a9ad2bf794a439ba2787985c34091decf53214b42034 not found: ID does not exist" containerID="a6ed2634bf8a97594ab7a9ad2bf794a439ba2787985c34091decf53214b42034" Oct 30 22:32:53.027463 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:53.027141 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6ed2634bf8a97594ab7a9ad2bf794a439ba2787985c34091decf53214b42034"} err="failed to get container status \"a6ed2634bf8a97594ab7a9ad2bf794a439ba2787985c34091decf53214b42034\": rpc error: code = NotFound desc = could not find container \"a6ed2634bf8a97594ab7a9ad2bf794a439ba2787985c34091decf53214b42034\": container with ID starting with a6ed2634bf8a97594ab7a9ad2bf794a439ba2787985c34091decf53214b42034 not found: ID does not exist" Oct 30 22:32:53.027463 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:53.027166 6006 scope.go:122] "RemoveContainer" containerID="947682e4a57dd01bffdc20aab0a087343fca49ca45f8637b1064679504cc061e" Oct 30 22:32:53.045441 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:53.042857 6006 scope.go:122] "RemoveContainer" containerID="947682e4a57dd01bffdc20aab0a087343fca49ca45f8637b1064679504cc061e" Oct 30 22:32:53.045441 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:32:53.043381 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"947682e4a57dd01bffdc20aab0a087343fca49ca45f8637b1064679504cc061e\": container with ID starting with 947682e4a57dd01bffdc20aab0a087343fca49ca45f8637b1064679504cc061e not found: ID does not exist" containerID="947682e4a57dd01bffdc20aab0a087343fca49ca45f8637b1064679504cc061e" Oct 30 22:32:53.045441 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:53.043427 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"947682e4a57dd01bffdc20aab0a087343fca49ca45f8637b1064679504cc061e"} err="failed to get container status \"947682e4a57dd01bffdc20aab0a087343fca49ca45f8637b1064679504cc061e\": rpc error: code = NotFound desc = could not find container \"947682e4a57dd01bffdc20aab0a087343fca49ca45f8637b1064679504cc061e\": container with ID starting with 947682e4a57dd01bffdc20aab0a087343fca49ca45f8637b1064679504cc061e not found: ID does not exist" Oct 30 22:32:53.216917 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:53.216824 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk"] Oct 30 22:32:53.222567 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:53.222531 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-bc714d/premium-w2-bc714d-kserve-755cdc9757-xqfvk"] Oct 30 22:32:53.338923 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:53.338887 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn"] Oct 30 22:32:53.344414 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:53.344382 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-bc714d/distinct-2-w2-bc714d-kserve-774cf84f99-cfnmn"] Oct 30 22:32:53.450929 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:53.450894 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569"] Oct 30 22:32:53.456175 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:53.456146 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-bc714d/simulator-w2-bc714d-kserve-cc58c74bd-68569"] Oct 30 22:32:53.603978 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:53.603942 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" path="/var/lib/kubelet/pods/08f28fb4-f013-4385-991d-742c9e38f4b2/volumes" Oct 30 22:32:53.604463 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:53.604448 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" path="/var/lib/kubelet/pods/a9fc3151-cd09-4454-89c0-dc57204054ca/volumes" Oct 30 22:32:53.604837 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:53.604824 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" path="/var/lib/kubelet/pods/fe28bccf-50cc-4c80-b29d-47810d3aabd2/volumes" Oct 30 22:32:57.910452 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:32:57.910409 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:32:57.999785 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:57.999747 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt"] Oct 30 22:32:58.045941 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:58.045903 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-c42c0a-ns4mt"] Oct 30 22:32:59.233842 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:59.233805 6006 scope.go:122] "RemoveContainer" containerID="53f6186bf9841e4fcdc3943d46575acfe88b954475613bf41508723b3629b40e" Oct 30 22:32:59.244304 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:59.244243 6006 scope.go:122] "RemoveContainer" containerID="9176da8d45fa1a7ef30ae42398a50c6af8765fd1a6a29b4e1266ba63d6a8d454" Oct 30 22:32:59.605150 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:32:59.605109 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbcf9f55-750b-4081-8c9c-636feed707e6" path="/var/lib/kubelet/pods/dbcf9f55-750b-4081-8c9c-636feed707e6/volumes" Oct 30 22:33:05.843018 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:05.842971 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk"] Oct 30 22:33:05.843692 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:05.843669 6006 cpu_manager.go:414] "containerMap: removing container" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" Oct 30 22:33:05.843790 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:05.843694 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" Oct 30 22:33:05.843790 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:05.843717 6006 cpu_manager.go:414] "containerMap: removing container" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" Oct 30 22:33:05.843790 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:05.843725 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" Oct 30 22:33:05.843790 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:05.843771 6006 cpu_manager.go:414] "containerMap: removing container" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" Oct 30 22:33:05.843790 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:05.843778 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" Oct 30 22:33:05.844044 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:05.843793 6006 cpu_manager.go:414] "containerMap: removing container" podUID="e9083308-566b-4535-b9f0-630f6775bce0" containerName="revoke-keys" Oct 30 22:33:05.844044 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:05.843802 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9083308-566b-4535-b9f0-630f6775bce0" containerName="revoke-keys" Oct 30 22:33:05.844044 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:05.843940 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f28fb4-f013-4385-991d-742c9e38f4b2" containerName="main" Oct 30 22:33:05.844044 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:05.843954 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9083308-566b-4535-b9f0-630f6775bce0" containerName="revoke-keys" Oct 30 22:33:05.844044 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:05.843969 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9fc3151-cd09-4454-89c0-dc57204054ca" containerName="main" Oct 30 22:33:05.844044 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:05.843978 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe28bccf-50cc-4c80-b29d-47810d3aabd2" containerName="main" Oct 30 22:33:05.849344 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:05.849320 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk" Oct 30 22:33:05.908387 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:05.908347 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk"] Oct 30 22:33:05.949160 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:05.949119 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:33:05.949160 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:05.949119 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:33:05.949458 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:05.949184 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:33:05.949458 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:05.949195 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:33:06.037367 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:06.037317 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd3ec6d4-0e2a-494d-908a-45080c2f9eb1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk\" (UID: \"dd3ec6d4-0e2a-494d-908a-45080c2f9eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk" Oct 30 22:33:06.037584 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:06.037494 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd3ec6d4-0e2a-494d-908a-45080c2f9eb1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk\" (UID: \"dd3ec6d4-0e2a-494d-908a-45080c2f9eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk" Oct 30 22:33:06.037584 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:06.037538 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb4df\" (UniqueName: \"kubernetes.io/projected/dd3ec6d4-0e2a-494d-908a-45080c2f9eb1-kube-api-access-lb4df\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk\" (UID: \"dd3ec6d4-0e2a-494d-908a-45080c2f9eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk" Oct 30 22:33:06.138573 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:06.138479 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd3ec6d4-0e2a-494d-908a-45080c2f9eb1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk\" (UID: \"dd3ec6d4-0e2a-494d-908a-45080c2f9eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk" Oct 30 22:33:06.138573 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:06.138534 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb4df\" (UniqueName: \"kubernetes.io/projected/dd3ec6d4-0e2a-494d-908a-45080c2f9eb1-kube-api-access-lb4df\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk\" (UID: \"dd3ec6d4-0e2a-494d-908a-45080c2f9eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk" Oct 30 22:33:06.138777 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:06.138619 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd3ec6d4-0e2a-494d-908a-45080c2f9eb1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk\" (UID: \"dd3ec6d4-0e2a-494d-908a-45080c2f9eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk" Oct 30 22:33:06.139346 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:06.139321 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd3ec6d4-0e2a-494d-908a-45080c2f9eb1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk\" (UID: \"dd3ec6d4-0e2a-494d-908a-45080c2f9eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk" Oct 30 22:33:06.141053 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:06.141025 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd3ec6d4-0e2a-494d-908a-45080c2f9eb1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk\" (UID: \"dd3ec6d4-0e2a-494d-908a-45080c2f9eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk" Oct 30 22:33:06.208345 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:06.208302 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb4df\" (UniqueName: \"kubernetes.io/projected/dd3ec6d4-0e2a-494d-908a-45080c2f9eb1-kube-api-access-lb4df\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk\" (UID: \"dd3ec6d4-0e2a-494d-908a-45080c2f9eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk" Oct 30 22:33:06.464689 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:06.464600 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk" Oct 30 22:33:06.841137 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:06.841095 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk"] Oct 30 22:33:06.842567 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:33:06.842542 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd3ec6d4_0e2a_494d_908a_45080c2f9eb1.slice/crio-3314eaf5d3d3d4c60b8b9fa6be6ce266b041e22093ce88a3c88bcabd602ce2a9 WatchSource:0}: Error finding container 3314eaf5d3d3d4c60b8b9fa6be6ce266b041e22093ce88a3c88bcabd602ce2a9: Status 404 returned error can't find the container with id 3314eaf5d3d3d4c60b8b9fa6be6ce266b041e22093ce88a3c88bcabd602ce2a9 Oct 30 22:33:07.090647 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:07.090549 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk" event={"ID":"dd3ec6d4-0e2a-494d-908a-45080c2f9eb1","Type":"ContainerStarted","Data":"3314eaf5d3d3d4c60b8b9fa6be6ce266b041e22093ce88a3c88bcabd602ce2a9"} Oct 30 22:33:08.098539 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:08.098503 6006 generic.go:356] "Generic (PLEG): container finished" podID="dd3ec6d4-0e2a-494d-908a-45080c2f9eb1" containerID="6c1fd63db5c7991fd246b0538604d221ef96f2a65ae743ade7fac65008ef7ece" exitCode=0 Oct 30 22:33:08.098937 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:08.098548 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk" event={"ID":"dd3ec6d4-0e2a-494d-908a-45080c2f9eb1","Type":"ContainerDied","Data":"6c1fd63db5c7991fd246b0538604d221ef96f2a65ae743ade7fac65008ef7ece"} Oct 30 22:33:09.259986 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:09.259958 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk" Oct 30 22:33:09.275438 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:09.275406 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd3ec6d4-0e2a-494d-908a-45080c2f9eb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd3ec6d4-0e2a-494d-908a-45080c2f9eb1-maas-api-xks-ca\") pod \"dd3ec6d4-0e2a-494d-908a-45080c2f9eb1\" (UID: \"dd3ec6d4-0e2a-494d-908a-45080c2f9eb1\") " Oct 30 22:33:09.275588 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:09.275538 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd3ec6d4-0e2a-494d-908a-45080c2f9eb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd3ec6d4-0e2a-494d-908a-45080c2f9eb1-maas-api-openshift-ca\") pod \"dd3ec6d4-0e2a-494d-908a-45080c2f9eb1\" (UID: \"dd3ec6d4-0e2a-494d-908a-45080c2f9eb1\") " Oct 30 22:33:09.275588 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:09.275577 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd3ec6d4-0e2a-494d-908a-45080c2f9eb1-kube-api-access-lb4df\" (UniqueName: \"kubernetes.io/projected/dd3ec6d4-0e2a-494d-908a-45080c2f9eb1-kube-api-access-lb4df\") pod \"dd3ec6d4-0e2a-494d-908a-45080c2f9eb1\" (UID: \"dd3ec6d4-0e2a-494d-908a-45080c2f9eb1\") " Oct 30 22:33:09.275932 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:09.275898 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd3ec6d4-0e2a-494d-908a-45080c2f9eb1-maas-api-openshift-ca" pod "dd3ec6d4-0e2a-494d-908a-45080c2f9eb1" (UID: "dd3ec6d4-0e2a-494d-908a-45080c2f9eb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:09.276050 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:09.276028 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd3ec6d4-0e2a-494d-908a-45080c2f9eb1-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:09.277512 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:09.277489 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd3ec6d4-0e2a-494d-908a-45080c2f9eb1-maas-api-xks-ca" pod "dd3ec6d4-0e2a-494d-908a-45080c2f9eb1" (UID: "dd3ec6d4-0e2a-494d-908a-45080c2f9eb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:09.277619 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:09.277581 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd3ec6d4-0e2a-494d-908a-45080c2f9eb1-kube-api-access-lb4df" pod "dd3ec6d4-0e2a-494d-908a-45080c2f9eb1" (UID: "dd3ec6d4-0e2a-494d-908a-45080c2f9eb1"). InnerVolumeSpecName "kube-api-access-lb4df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:09.377062 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:09.376978 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd3ec6d4-0e2a-494d-908a-45080c2f9eb1-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:09.377062 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:09.377018 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb4df\" (UniqueName: \"kubernetes.io/projected/dd3ec6d4-0e2a-494d-908a-45080c2f9eb1-kube-api-access-lb4df\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:10.115740 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:10.115692 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk" Oct 30 22:33:10.115953 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:10.115710 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk" event={"ID":"dd3ec6d4-0e2a-494d-908a-45080c2f9eb1","Type":"ContainerDied","Data":"3314eaf5d3d3d4c60b8b9fa6be6ce266b041e22093ce88a3c88bcabd602ce2a9"} Oct 30 22:33:10.115953 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:10.115890 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3314eaf5d3d3d4c60b8b9fa6be6ce266b041e22093ce88a3c88bcabd602ce2a9" Oct 30 22:33:13.843623 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:13.843581 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:33:16.188191 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:16.188151 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m"] Oct 30 22:33:16.288929 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:16.288890 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7"] Oct 30 22:33:17.136473 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.136427 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc37f9f4-7mzcb"] Oct 30 22:33:17.137188 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.137155 6006 cpu_manager.go:414] "containerMap: removing container" podUID="dd3ec6d4-0e2a-494d-908a-45080c2f9eb1" containerName="revoke-keys" Oct 30 22:33:17.137188 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.137176 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd3ec6d4-0e2a-494d-908a-45080c2f9eb1" containerName="revoke-keys" Oct 30 22:33:17.137451 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.137311 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd3ec6d4-0e2a-494d-908a-45080c2f9eb1" containerName="revoke-keys" Oct 30 22:33:17.141416 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.141391 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc37f9f4-7mzcb" Oct 30 22:33:17.154083 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.154056 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86e37fb2-tpm9t"] Oct 30 22:33:17.158304 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.158280 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e37fb2-tpm9t" Oct 30 22:33:17.191175 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.191136 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:33:17.191175 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.191136 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:33:17.191630 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.191358 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:33:17.193089 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.193067 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:33:17.206203 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.206167 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc37f9f4-7mzcb"] Oct 30 22:33:17.215524 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.215489 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86e37fb2-tpm9t"] Oct 30 22:33:17.274033 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.273995 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4adb785-11b7-44a1-a95d-c5ecac657922-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc37f9f4-7mzcb\" (UID: \"b4adb785-11b7-44a1-a95d-c5ecac657922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc37f9f4-7mzcb" Oct 30 22:33:17.274217 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.274057 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4adb785-11b7-44a1-a95d-c5ecac657922-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc37f9f4-7mzcb\" (UID: \"b4adb785-11b7-44a1-a95d-c5ecac657922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc37f9f4-7mzcb" Oct 30 22:33:17.274217 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.274127 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16f728f0-4dce-43c2-8e6d-c2c38ee44c63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86e37fb2-tpm9t\" (UID: \"16f728f0-4dce-43c2-8e6d-c2c38ee44c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e37fb2-tpm9t" Oct 30 22:33:17.274217 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.274189 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16f728f0-4dce-43c2-8e6d-c2c38ee44c63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86e37fb2-tpm9t\" (UID: \"16f728f0-4dce-43c2-8e6d-c2c38ee44c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e37fb2-tpm9t" Oct 30 22:33:17.274399 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.274224 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7phv5\" (UniqueName: \"kubernetes.io/projected/b4adb785-11b7-44a1-a95d-c5ecac657922-kube-api-access-7phv5\") pod \"maas-api-revoke-sub-cc37f9f4-7mzcb\" (UID: \"b4adb785-11b7-44a1-a95d-c5ecac657922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc37f9f4-7mzcb" Oct 30 22:33:17.274399 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.274262 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-889vr\" (UniqueName: \"kubernetes.io/projected/16f728f0-4dce-43c2-8e6d-c2c38ee44c63-kube-api-access-889vr\") pod \"maas-api-revoke-sub-86e37fb2-tpm9t\" (UID: \"16f728f0-4dce-43c2-8e6d-c2c38ee44c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e37fb2-tpm9t" Oct 30 22:33:17.375274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.375228 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16f728f0-4dce-43c2-8e6d-c2c38ee44c63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86e37fb2-tpm9t\" (UID: \"16f728f0-4dce-43c2-8e6d-c2c38ee44c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e37fb2-tpm9t" Oct 30 22:33:17.375443 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.375325 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16f728f0-4dce-43c2-8e6d-c2c38ee44c63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86e37fb2-tpm9t\" (UID: \"16f728f0-4dce-43c2-8e6d-c2c38ee44c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e37fb2-tpm9t" Oct 30 22:33:17.375443 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.375384 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7phv5\" (UniqueName: \"kubernetes.io/projected/b4adb785-11b7-44a1-a95d-c5ecac657922-kube-api-access-7phv5\") pod \"maas-api-revoke-sub-cc37f9f4-7mzcb\" (UID: \"b4adb785-11b7-44a1-a95d-c5ecac657922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc37f9f4-7mzcb" Oct 30 22:33:17.375443 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.375415 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-889vr\" (UniqueName: \"kubernetes.io/projected/16f728f0-4dce-43c2-8e6d-c2c38ee44c63-kube-api-access-889vr\") pod \"maas-api-revoke-sub-86e37fb2-tpm9t\" (UID: \"16f728f0-4dce-43c2-8e6d-c2c38ee44c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e37fb2-tpm9t" Oct 30 22:33:17.375608 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.375566 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4adb785-11b7-44a1-a95d-c5ecac657922-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc37f9f4-7mzcb\" (UID: \"b4adb785-11b7-44a1-a95d-c5ecac657922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc37f9f4-7mzcb" Oct 30 22:33:17.375678 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.375618 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4adb785-11b7-44a1-a95d-c5ecac657922-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc37f9f4-7mzcb\" (UID: \"b4adb785-11b7-44a1-a95d-c5ecac657922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc37f9f4-7mzcb" Oct 30 22:33:17.376237 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.376203 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16f728f0-4dce-43c2-8e6d-c2c38ee44c63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86e37fb2-tpm9t\" (UID: \"16f728f0-4dce-43c2-8e6d-c2c38ee44c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e37fb2-tpm9t" Oct 30 22:33:17.376382 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.376210 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4adb785-11b7-44a1-a95d-c5ecac657922-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc37f9f4-7mzcb\" (UID: \"b4adb785-11b7-44a1-a95d-c5ecac657922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc37f9f4-7mzcb" Oct 30 22:33:17.377890 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.377840 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16f728f0-4dce-43c2-8e6d-c2c38ee44c63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86e37fb2-tpm9t\" (UID: \"16f728f0-4dce-43c2-8e6d-c2c38ee44c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e37fb2-tpm9t" Oct 30 22:33:17.378003 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.377914 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4adb785-11b7-44a1-a95d-c5ecac657922-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc37f9f4-7mzcb\" (UID: \"b4adb785-11b7-44a1-a95d-c5ecac657922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc37f9f4-7mzcb" Oct 30 22:33:17.438017 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.437938 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7phv5\" (UniqueName: \"kubernetes.io/projected/b4adb785-11b7-44a1-a95d-c5ecac657922-kube-api-access-7phv5\") pod \"maas-api-revoke-sub-cc37f9f4-7mzcb\" (UID: \"b4adb785-11b7-44a1-a95d-c5ecac657922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc37f9f4-7mzcb" Oct 30 22:33:17.438194 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.438100 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-889vr\" (UniqueName: \"kubernetes.io/projected/16f728f0-4dce-43c2-8e6d-c2c38ee44c63-kube-api-access-889vr\") pod \"maas-api-revoke-sub-86e37fb2-tpm9t\" (UID: \"16f728f0-4dce-43c2-8e6d-c2c38ee44c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e37fb2-tpm9t" Oct 30 22:33:17.457108 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.457030 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc37f9f4-7mzcb" Oct 30 22:33:17.471945 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.471922 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e37fb2-tpm9t" Oct 30 22:33:17.850569 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.850537 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc37f9f4-7mzcb"] Oct 30 22:33:17.851877 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:33:17.851846 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4adb785_11b7_44a1_a95d_c5ecac657922.slice/crio-6ba66889d37f526429c494f25c32920d89640363396bb985f553b5f5e7dc96e7 WatchSource:0}: Error finding container 6ba66889d37f526429c494f25c32920d89640363396bb985f553b5f5e7dc96e7: Status 404 returned error can't find the container with id 6ba66889d37f526429c494f25c32920d89640363396bb985f553b5f5e7dc96e7 Oct 30 22:33:17.909495 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:17.906459 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86e37fb2-tpm9t"] Oct 30 22:33:17.912969 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:33:17.912908 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f728f0_4dce_43c2_8e6d_c2c38ee44c63.slice/crio-44be4e0367af61c8c0e149624bc0cf5da600ddd37a819f1ec36a7d31e2501f18 WatchSource:0}: Error finding container 44be4e0367af61c8c0e149624bc0cf5da600ddd37a819f1ec36a7d31e2501f18: Status 404 returned error can't find the container with id 44be4e0367af61c8c0e149624bc0cf5da600ddd37a819f1ec36a7d31e2501f18 Oct 30 22:33:18.195857 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:18.195815 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc37f9f4-7mzcb" event={"ID":"b4adb785-11b7-44a1-a95d-c5ecac657922","Type":"ContainerStarted","Data":"6ba66889d37f526429c494f25c32920d89640363396bb985f553b5f5e7dc96e7"} Oct 30 22:33:18.197187 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:18.197160 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e37fb2-tpm9t" event={"ID":"16f728f0-4dce-43c2-8e6d-c2c38ee44c63","Type":"ContainerStarted","Data":"44be4e0367af61c8c0e149624bc0cf5da600ddd37a819f1ec36a7d31e2501f18"} Oct 30 22:33:19.217921 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:19.217829 6006 generic.go:356] "Generic (PLEG): container finished" podID="16f728f0-4dce-43c2-8e6d-c2c38ee44c63" containerID="732718fcd2aade34aebb8e1aee8c287291712619cf6eac064ac1e6306ddbdfce" exitCode=0 Oct 30 22:33:19.218428 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:19.217930 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e37fb2-tpm9t" event={"ID":"16f728f0-4dce-43c2-8e6d-c2c38ee44c63","Type":"ContainerDied","Data":"732718fcd2aade34aebb8e1aee8c287291712619cf6eac064ac1e6306ddbdfce"} Oct 30 22:33:19.220025 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:19.219996 6006 generic.go:356] "Generic (PLEG): container finished" podID="b4adb785-11b7-44a1-a95d-c5ecac657922" containerID="c98923e0aa69c3884248595b8dd57921fc2f9d91e261419e376acaba078eec78" exitCode=0 Oct 30 22:33:19.220136 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:19.220076 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc37f9f4-7mzcb" event={"ID":"b4adb785-11b7-44a1-a95d-c5ecac657922","Type":"ContainerDied","Data":"c98923e0aa69c3884248595b8dd57921fc2f9d91e261419e376acaba078eec78"} Oct 30 22:33:20.392894 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:20.392865 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc37f9f4-7mzcb" Oct 30 22:33:20.430943 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:20.430911 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e37fb2-tpm9t" Oct 30 22:33:20.529338 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:20.529211 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16f728f0-4dce-43c2-8e6d-c2c38ee44c63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16f728f0-4dce-43c2-8e6d-c2c38ee44c63-maas-api-openshift-ca\") pod \"16f728f0-4dce-43c2-8e6d-c2c38ee44c63\" (UID: \"16f728f0-4dce-43c2-8e6d-c2c38ee44c63\") " Oct 30 22:33:20.529338 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:20.529281 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4adb785-11b7-44a1-a95d-c5ecac657922-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4adb785-11b7-44a1-a95d-c5ecac657922-maas-api-xks-ca\") pod \"b4adb785-11b7-44a1-a95d-c5ecac657922\" (UID: \"b4adb785-11b7-44a1-a95d-c5ecac657922\") " Oct 30 22:33:20.529586 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:20.529367 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16f728f0-4dce-43c2-8e6d-c2c38ee44c63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16f728f0-4dce-43c2-8e6d-c2c38ee44c63-maas-api-xks-ca\") pod \"16f728f0-4dce-43c2-8e6d-c2c38ee44c63\" (UID: \"16f728f0-4dce-43c2-8e6d-c2c38ee44c63\") " Oct 30 22:33:20.529586 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:20.529404 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4adb785-11b7-44a1-a95d-c5ecac657922-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4adb785-11b7-44a1-a95d-c5ecac657922-maas-api-openshift-ca\") pod \"b4adb785-11b7-44a1-a95d-c5ecac657922\" (UID: \"b4adb785-11b7-44a1-a95d-c5ecac657922\") " Oct 30 22:33:20.529586 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:20.529434 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4adb785-11b7-44a1-a95d-c5ecac657922-kube-api-access-7phv5\" (UniqueName: \"kubernetes.io/projected/b4adb785-11b7-44a1-a95d-c5ecac657922-kube-api-access-7phv5\") pod \"b4adb785-11b7-44a1-a95d-c5ecac657922\" (UID: \"b4adb785-11b7-44a1-a95d-c5ecac657922\") " Oct 30 22:33:20.529586 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:20.529533 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16f728f0-4dce-43c2-8e6d-c2c38ee44c63-kube-api-access-889vr\" (UniqueName: \"kubernetes.io/projected/16f728f0-4dce-43c2-8e6d-c2c38ee44c63-kube-api-access-889vr\") pod \"16f728f0-4dce-43c2-8e6d-c2c38ee44c63\" (UID: \"16f728f0-4dce-43c2-8e6d-c2c38ee44c63\") " Oct 30 22:33:20.529794 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:20.529699 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16f728f0-4dce-43c2-8e6d-c2c38ee44c63-maas-api-openshift-ca" pod "16f728f0-4dce-43c2-8e6d-c2c38ee44c63" (UID: "16f728f0-4dce-43c2-8e6d-c2c38ee44c63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:20.530069 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:20.530035 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16f728f0-4dce-43c2-8e6d-c2c38ee44c63-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:20.530069 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:20.530044 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4adb785-11b7-44a1-a95d-c5ecac657922-maas-api-openshift-ca" pod "b4adb785-11b7-44a1-a95d-c5ecac657922" (UID: "b4adb785-11b7-44a1-a95d-c5ecac657922"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:20.531962 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:20.531933 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16f728f0-4dce-43c2-8e6d-c2c38ee44c63-maas-api-xks-ca" pod "16f728f0-4dce-43c2-8e6d-c2c38ee44c63" (UID: "16f728f0-4dce-43c2-8e6d-c2c38ee44c63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:20.532212 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:20.532190 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4adb785-11b7-44a1-a95d-c5ecac657922-maas-api-xks-ca" pod "b4adb785-11b7-44a1-a95d-c5ecac657922" (UID: "b4adb785-11b7-44a1-a95d-c5ecac657922"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:20.532464 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:20.532438 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4adb785-11b7-44a1-a95d-c5ecac657922-kube-api-access-7phv5" pod "b4adb785-11b7-44a1-a95d-c5ecac657922" (UID: "b4adb785-11b7-44a1-a95d-c5ecac657922"). InnerVolumeSpecName "kube-api-access-7phv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:20.532719 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:20.532693 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16f728f0-4dce-43c2-8e6d-c2c38ee44c63-kube-api-access-889vr" pod "16f728f0-4dce-43c2-8e6d-c2c38ee44c63" (UID: "16f728f0-4dce-43c2-8e6d-c2c38ee44c63"). InnerVolumeSpecName "kube-api-access-889vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:20.631215 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:20.631159 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4adb785-11b7-44a1-a95d-c5ecac657922-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:20.631215 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:20.631198 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16f728f0-4dce-43c2-8e6d-c2c38ee44c63-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:20.631215 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:20.631214 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4adb785-11b7-44a1-a95d-c5ecac657922-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:20.631490 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:20.631231 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7phv5\" (UniqueName: \"kubernetes.io/projected/b4adb785-11b7-44a1-a95d-c5ecac657922-kube-api-access-7phv5\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:20.631490 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:20.631260 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-889vr\" (UniqueName: \"kubernetes.io/projected/16f728f0-4dce-43c2-8e6d-c2c38ee44c63-kube-api-access-889vr\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:21.189337 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:21.189288 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m" podUID="d3dc9f55-2232-45df-bff5-e307125462de" containerName="payload-processing" containerID="cri-o://dd9a95e92a563940496750a563e39a72678794494f34c35ac20280865a88f525" gracePeriod=25 Oct 30 22:33:21.216452 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:33:21.216423 6006 logging.go:55] [core] [Channel #3224 SubChannel #3225]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.127:50052", ServerName: "10.132.0.127:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.127:50052: connect: connection refused" Oct 30 22:33:21.239150 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:21.239118 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc37f9f4-7mzcb" Oct 30 22:33:21.239396 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:21.239146 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc37f9f4-7mzcb" event={"ID":"b4adb785-11b7-44a1-a95d-c5ecac657922","Type":"ContainerDied","Data":"6ba66889d37f526429c494f25c32920d89640363396bb985f553b5f5e7dc96e7"} Oct 30 22:33:21.239396 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:21.239199 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ba66889d37f526429c494f25c32920d89640363396bb985f553b5f5e7dc96e7" Oct 30 22:33:21.240869 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:21.240846 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e37fb2-tpm9t" Oct 30 22:33:21.240982 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:21.240849 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e37fb2-tpm9t" event={"ID":"16f728f0-4dce-43c2-8e6d-c2c38ee44c63","Type":"ContainerDied","Data":"44be4e0367af61c8c0e149624bc0cf5da600ddd37a819f1ec36a7d31e2501f18"} Oct 30 22:33:21.240982 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:21.240958 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44be4e0367af61c8c0e149624bc0cf5da600ddd37a819f1ec36a7d31e2501f18" Oct 30 22:33:21.289530 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:21.289481 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7" podUID="5c5e3d88-1dfc-476c-8f88-bf35edbbc82a" containerName="payload-pre-processing" containerID="cri-o://2bf51d6752d6c96206dd0c92f4570ef609c8a0179a6e43db92d59fcd51b76b35" gracePeriod=25 Oct 30 22:33:21.464633 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:21.464606 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m" Oct 30 22:33:21.541342 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:21.541296 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3dc9f55-2232-45df-bff5-e307125462de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3dc9f55-2232-45df-bff5-e307125462de-plugins-config-volume\") pod \"d3dc9f55-2232-45df-bff5-e307125462de\" (UID: \"d3dc9f55-2232-45df-bff5-e307125462de\") " Oct 30 22:33:21.541583 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:21.541552 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3dc9f55-2232-45df-bff5-e307125462de-kube-api-access-ftt4v\" (UniqueName: \"kubernetes.io/projected/d3dc9f55-2232-45df-bff5-e307125462de-kube-api-access-ftt4v\") pod \"d3dc9f55-2232-45df-bff5-e307125462de\" (UID: \"d3dc9f55-2232-45df-bff5-e307125462de\") " Oct 30 22:33:21.541746 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:21.541721 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3dc9f55-2232-45df-bff5-e307125462de-plugins-config-volume" pod "d3dc9f55-2232-45df-bff5-e307125462de" (UID: "d3dc9f55-2232-45df-bff5-e307125462de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:21.542469 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:21.542439 6006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3dc9f55-2232-45df-bff5-e307125462de-plugins-config-volume\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:21.543829 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:21.543797 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3dc9f55-2232-45df-bff5-e307125462de-kube-api-access-ftt4v" pod "d3dc9f55-2232-45df-bff5-e307125462de" (UID: "d3dc9f55-2232-45df-bff5-e307125462de"). InnerVolumeSpecName "kube-api-access-ftt4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:21.571626 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:21.571600 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7" Oct 30 22:33:21.642830 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:21.642798 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c5e3d88-1dfc-476c-8f88-bf35edbbc82a-kube-api-access-5txcr\" (UniqueName: \"kubernetes.io/projected/5c5e3d88-1dfc-476c-8f88-bf35edbbc82a-kube-api-access-5txcr\") pod \"5c5e3d88-1dfc-476c-8f88-bf35edbbc82a\" (UID: \"5c5e3d88-1dfc-476c-8f88-bf35edbbc82a\") " Oct 30 22:33:21.643060 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:21.642866 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c5e3d88-1dfc-476c-8f88-bf35edbbc82a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5e3d88-1dfc-476c-8f88-bf35edbbc82a-plugins-config-volume\") pod \"5c5e3d88-1dfc-476c-8f88-bf35edbbc82a\" (UID: \"5c5e3d88-1dfc-476c-8f88-bf35edbbc82a\") " Oct 30 22:33:21.643368 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:21.643348 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftt4v\" (UniqueName: \"kubernetes.io/projected/d3dc9f55-2232-45df-bff5-e307125462de-kube-api-access-ftt4v\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:21.643368 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:21.643355 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c5e3d88-1dfc-476c-8f88-bf35edbbc82a-plugins-config-volume" pod "5c5e3d88-1dfc-476c-8f88-bf35edbbc82a" (UID: "5c5e3d88-1dfc-476c-8f88-bf35edbbc82a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:21.645207 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:21.645182 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c5e3d88-1dfc-476c-8f88-bf35edbbc82a-kube-api-access-5txcr" pod "5c5e3d88-1dfc-476c-8f88-bf35edbbc82a" (UID: "5c5e3d88-1dfc-476c-8f88-bf35edbbc82a"). InnerVolumeSpecName "kube-api-access-5txcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:21.745160 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:21.745082 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5txcr\" (UniqueName: \"kubernetes.io/projected/5c5e3d88-1dfc-476c-8f88-bf35edbbc82a-kube-api-access-5txcr\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:21.745160 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:21.745120 6006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5e3d88-1dfc-476c-8f88-bf35edbbc82a-plugins-config-volume\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:22.216894 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:22.216837 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m" podUID="d3dc9f55-2232-45df-bff5-e307125462de" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.127:50052\" within 1s: context deadline exceeded" Oct 30 22:33:22.249226 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:22.249189 6006 generic.go:356] "Generic (PLEG): container finished" podID="d3dc9f55-2232-45df-bff5-e307125462de" containerID="dd9a95e92a563940496750a563e39a72678794494f34c35ac20280865a88f525" exitCode=0 Oct 30 22:33:22.249418 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:22.249276 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m" Oct 30 22:33:22.249418 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:22.249278 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m" event={"ID":"d3dc9f55-2232-45df-bff5-e307125462de","Type":"ContainerDied","Data":"dd9a95e92a563940496750a563e39a72678794494f34c35ac20280865a88f525"} Oct 30 22:33:22.249513 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:22.249439 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m" event={"ID":"d3dc9f55-2232-45df-bff5-e307125462de","Type":"ContainerDied","Data":"1750d60d18e14776f5d71003e808f51ab632b340e7fa7d4ab4976d964cd4dc13"} Oct 30 22:33:22.249513 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:22.249465 6006 scope.go:122] "RemoveContainer" containerID="dd9a95e92a563940496750a563e39a72678794494f34c35ac20280865a88f525" Oct 30 22:33:22.251181 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:22.251157 6006 generic.go:356] "Generic (PLEG): container finished" podID="5c5e3d88-1dfc-476c-8f88-bf35edbbc82a" containerID="2bf51d6752d6c96206dd0c92f4570ef609c8a0179a6e43db92d59fcd51b76b35" exitCode=0 Oct 30 22:33:22.251341 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:22.251205 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7" Oct 30 22:33:22.251341 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:22.251215 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7" event={"ID":"5c5e3d88-1dfc-476c-8f88-bf35edbbc82a","Type":"ContainerDied","Data":"2bf51d6752d6c96206dd0c92f4570ef609c8a0179a6e43db92d59fcd51b76b35"} Oct 30 22:33:22.252801 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:22.251628 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7" event={"ID":"5c5e3d88-1dfc-476c-8f88-bf35edbbc82a","Type":"ContainerDied","Data":"8563f27ef41364f6968813e2ed2ed74ca37b45c6acfd61ba6e50c038d3a56f14"} Oct 30 22:33:22.262047 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:22.262025 6006 scope.go:122] "RemoveContainer" containerID="dd9a95e92a563940496750a563e39a72678794494f34c35ac20280865a88f525" Oct 30 22:33:22.262388 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:33:22.262361 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd9a95e92a563940496750a563e39a72678794494f34c35ac20280865a88f525\": container with ID starting with dd9a95e92a563940496750a563e39a72678794494f34c35ac20280865a88f525 not found: ID does not exist" containerID="dd9a95e92a563940496750a563e39a72678794494f34c35ac20280865a88f525" Oct 30 22:33:22.262484 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:22.262397 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd9a95e92a563940496750a563e39a72678794494f34c35ac20280865a88f525"} err="failed to get container status \"dd9a95e92a563940496750a563e39a72678794494f34c35ac20280865a88f525\": rpc error: code = NotFound desc = could not find container \"dd9a95e92a563940496750a563e39a72678794494f34c35ac20280865a88f525\": container with ID starting with dd9a95e92a563940496750a563e39a72678794494f34c35ac20280865a88f525 not found: ID does not exist" Oct 30 22:33:22.262484 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:22.262422 6006 scope.go:122] "RemoveContainer" containerID="2bf51d6752d6c96206dd0c92f4570ef609c8a0179a6e43db92d59fcd51b76b35" Oct 30 22:33:22.273844 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:22.273820 6006 scope.go:122] "RemoveContainer" containerID="2bf51d6752d6c96206dd0c92f4570ef609c8a0179a6e43db92d59fcd51b76b35" Oct 30 22:33:22.274209 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:33:22.274187 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf51d6752d6c96206dd0c92f4570ef609c8a0179a6e43db92d59fcd51b76b35\": container with ID starting with 2bf51d6752d6c96206dd0c92f4570ef609c8a0179a6e43db92d59fcd51b76b35 not found: ID does not exist" containerID="2bf51d6752d6c96206dd0c92f4570ef609c8a0179a6e43db92d59fcd51b76b35" Oct 30 22:33:22.274326 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:22.274219 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf51d6752d6c96206dd0c92f4570ef609c8a0179a6e43db92d59fcd51b76b35"} err="failed to get container status \"2bf51d6752d6c96206dd0c92f4570ef609c8a0179a6e43db92d59fcd51b76b35\": rpc error: code = NotFound desc = could not find container \"2bf51d6752d6c96206dd0c92f4570ef609c8a0179a6e43db92d59fcd51b76b35\": container with ID starting with 2bf51d6752d6c96206dd0c92f4570ef609c8a0179a6e43db92d59fcd51b76b35 not found: ID does not exist" Oct 30 22:33:22.411664 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:22.411625 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m"] Oct 30 22:33:22.415572 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:22.415543 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-ed44da-5dfdf86775-dsj5m"] Oct 30 22:33:22.517242 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:22.517136 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7"] Oct 30 22:33:22.521478 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:22.521448 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ed44da-887755447-26dm7"] Oct 30 22:33:23.603560 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:23.603521 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c5e3d88-1dfc-476c-8f88-bf35edbbc82a" path="/var/lib/kubelet/pods/5c5e3d88-1dfc-476c-8f88-bf35edbbc82a/volumes" Oct 30 22:33:23.604042 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:23.603940 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3dc9f55-2232-45df-bff5-e307125462de" path="/var/lib/kubelet/pods/d3dc9f55-2232-45df-bff5-e307125462de/volumes" Oct 30 22:33:27.912610 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:33:27.912554 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:33:35.097070 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:35.096967 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33c6d78b-pdh26"] Oct 30 22:33:35.099811 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:35.099777 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0feb5925-f7fh9"] Oct 30 22:33:35.104931 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:35.104899 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33c6d78b-pdh26"] Oct 30 22:33:35.106934 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:35.106911 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0feb5925-f7fh9"] Oct 30 22:33:35.611506 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:35.611468 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3094773b-d617-49d9-ac6a-435bd081ff21" path="/var/lib/kubelet/pods/3094773b-d617-49d9-ac6a-435bd081ff21/volumes" Oct 30 22:33:35.611921 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:35.611907 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26" path="/var/lib/kubelet/pods/dc04d6d3-5b6b-4a09-a0c0-d8f41900ec26/volumes" Oct 30 22:33:49.061140 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:49.061099 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:33:57.910962 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:33:57.910907 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:33:59.302177 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.302140 6006 scope.go:122] "RemoveContainer" containerID="5a314bdc8e01fd3b160c15712f9453430c15279c2a526395baf5c2c3cc9e9a7a" Oct 30 22:33:59.314990 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.314960 6006 scope.go:122] "RemoveContainer" containerID="bb4705bb52c8a7fd275dd3a84552fa8d54ff4590d1bda5fccf4db60abe8e1ff1" Oct 30 22:33:59.327559 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.327531 6006 scope.go:122] "RemoveContainer" containerID="ab74db2b9aa52ec51cd0e1e4a4fac42e03663f8a7ae36019e35ec0f9e8abf565" Oct 30 22:33:59.339805 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.339777 6006 scope.go:122] "RemoveContainer" containerID="d8a42bf2cdd274afb0b5cc456248b26889927fce9c8123cef5b8ae850f7ecc52" Oct 30 22:33:59.748190 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.748108 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz"] Oct 30 22:33:59.748803 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.748775 6006 cpu_manager.go:414] "containerMap: removing container" podUID="16f728f0-4dce-43c2-8e6d-c2c38ee44c63" containerName="revoke-keys" Oct 30 22:33:59.748803 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.748800 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16f728f0-4dce-43c2-8e6d-c2c38ee44c63" containerName="revoke-keys" Oct 30 22:33:59.749033 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.748834 6006 cpu_manager.go:414] "containerMap: removing container" podUID="5c5e3d88-1dfc-476c-8f88-bf35edbbc82a" containerName="payload-pre-processing" Oct 30 22:33:59.749033 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.748842 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c5e3d88-1dfc-476c-8f88-bf35edbbc82a" containerName="payload-pre-processing" Oct 30 22:33:59.749033 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.748859 6006 cpu_manager.go:414] "containerMap: removing container" podUID="b4adb785-11b7-44a1-a95d-c5ecac657922" containerName="revoke-keys" Oct 30 22:33:59.749033 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.748866 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4adb785-11b7-44a1-a95d-c5ecac657922" containerName="revoke-keys" Oct 30 22:33:59.749033 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.748884 6006 cpu_manager.go:414] "containerMap: removing container" podUID="d3dc9f55-2232-45df-bff5-e307125462de" containerName="payload-processing" Oct 30 22:33:59.749033 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.748892 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3dc9f55-2232-45df-bff5-e307125462de" containerName="payload-processing" Oct 30 22:33:59.749033 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.749008 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3dc9f55-2232-45df-bff5-e307125462de" containerName="payload-processing" Oct 30 22:33:59.749033 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.749027 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="16f728f0-4dce-43c2-8e6d-c2c38ee44c63" containerName="revoke-keys" Oct 30 22:33:59.749399 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.749041 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c5e3d88-1dfc-476c-8f88-bf35edbbc82a" containerName="payload-pre-processing" Oct 30 22:33:59.749399 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.749049 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4adb785-11b7-44a1-a95d-c5ecac657922" containerName="revoke-keys" Oct 30 22:33:59.754220 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.754193 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" Oct 30 22:33:59.787518 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.787474 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-dshm\") pod \"test-model-75fb62-kserve-b4cb96d67-frtsz\" (UID: \"772ecd37-9f05-48de-91e3-0ba670519f00\") " pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" Oct 30 22:33:59.787518 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.787515 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-home\") pod \"test-model-75fb62-kserve-b4cb96d67-frtsz\" (UID: \"772ecd37-9f05-48de-91e3-0ba670519f00\") " pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" Oct 30 22:33:59.787762 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.787634 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfpk5\" (UniqueName: \"kubernetes.io/projected/772ecd37-9f05-48de-91e3-0ba670519f00-kube-api-access-kfpk5\") pod \"test-model-75fb62-kserve-b4cb96d67-frtsz\" (UID: \"772ecd37-9f05-48de-91e3-0ba670519f00\") " pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" Oct 30 22:33:59.787762 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.787676 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-tmp-dir\") pod \"test-model-75fb62-kserve-b4cb96d67-frtsz\" (UID: \"772ecd37-9f05-48de-91e3-0ba670519f00\") " pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" Oct 30 22:33:59.787839 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.787798 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-model-cache\") pod \"test-model-75fb62-kserve-b4cb96d67-frtsz\" (UID: \"772ecd37-9f05-48de-91e3-0ba670519f00\") " pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" Oct 30 22:33:59.787839 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.787821 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772ecd37-9f05-48de-91e3-0ba670519f00-tls-certs\") pod \"test-model-75fb62-kserve-b4cb96d67-frtsz\" (UID: \"772ecd37-9f05-48de-91e3-0ba670519f00\") " pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" Oct 30 22:33:59.853892 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.853845 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-75fb62\"/\"kube-root-ca.crt\"" Oct 30 22:33:59.853892 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.853845 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-75fb62\"/\"openshift-service-ca.crt\"" Oct 30 22:33:59.854146 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.853986 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-75fb62\"/\"default-dockercfg-zh48t\"" Oct 30 22:33:59.854498 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.854465 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-75fb62\"/\"test-model-75fb62-kserve-self-signed-certs\"" Oct 30 22:33:59.866100 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.859098 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz"] Oct 30 22:33:59.890179 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.890127 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfpk5\" (UniqueName: \"kubernetes.io/projected/772ecd37-9f05-48de-91e3-0ba670519f00-kube-api-access-kfpk5\") pod \"test-model-75fb62-kserve-b4cb96d67-frtsz\" (UID: \"772ecd37-9f05-48de-91e3-0ba670519f00\") " pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" Oct 30 22:33:59.890443 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.890190 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-tmp-dir\") pod \"test-model-75fb62-kserve-b4cb96d67-frtsz\" (UID: \"772ecd37-9f05-48de-91e3-0ba670519f00\") " pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" Oct 30 22:33:59.890443 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.890312 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-model-cache\") pod \"test-model-75fb62-kserve-b4cb96d67-frtsz\" (UID: \"772ecd37-9f05-48de-91e3-0ba670519f00\") " pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" Oct 30 22:33:59.890443 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.890342 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772ecd37-9f05-48de-91e3-0ba670519f00-tls-certs\") pod \"test-model-75fb62-kserve-b4cb96d67-frtsz\" (UID: \"772ecd37-9f05-48de-91e3-0ba670519f00\") " pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" Oct 30 22:33:59.890443 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.890409 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-dshm\") pod \"test-model-75fb62-kserve-b4cb96d67-frtsz\" (UID: \"772ecd37-9f05-48de-91e3-0ba670519f00\") " pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" Oct 30 22:33:59.890669 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.890450 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-home\") pod \"test-model-75fb62-kserve-b4cb96d67-frtsz\" (UID: \"772ecd37-9f05-48de-91e3-0ba670519f00\") " pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" Oct 30 22:33:59.891188 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.891144 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-model-cache\") pod \"test-model-75fb62-kserve-b4cb96d67-frtsz\" (UID: \"772ecd37-9f05-48de-91e3-0ba670519f00\") " pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" Oct 30 22:33:59.892319 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.892191 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-tmp-dir\") pod \"test-model-75fb62-kserve-b4cb96d67-frtsz\" (UID: \"772ecd37-9f05-48de-91e3-0ba670519f00\") " pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" Oct 30 22:33:59.914436 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.913938 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772ecd37-9f05-48de-91e3-0ba670519f00-tls-certs\") pod \"test-model-75fb62-kserve-b4cb96d67-frtsz\" (UID: \"772ecd37-9f05-48de-91e3-0ba670519f00\") " pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" Oct 30 22:33:59.923623 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.923577 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-home\") pod \"test-model-75fb62-kserve-b4cb96d67-frtsz\" (UID: \"772ecd37-9f05-48de-91e3-0ba670519f00\") " pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" Oct 30 22:33:59.937581 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.937549 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-dshm\") pod \"test-model-75fb62-kserve-b4cb96d67-frtsz\" (UID: \"772ecd37-9f05-48de-91e3-0ba670519f00\") " pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" Oct 30 22:33:59.967387 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:33:59.967308 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfpk5\" (UniqueName: \"kubernetes.io/projected/772ecd37-9f05-48de-91e3-0ba670519f00-kube-api-access-kfpk5\") pod \"test-model-75fb62-kserve-b4cb96d67-frtsz\" (UID: \"772ecd37-9f05-48de-91e3-0ba670519f00\") " pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" Oct 30 22:34:00.067180 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:00.067131 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" Oct 30 22:34:00.478161 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:00.469900 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz"] Oct 30 22:34:00.527382 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:00.527338 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" event={"ID":"772ecd37-9f05-48de-91e3-0ba670519f00","Type":"ContainerStarted","Data":"c2d7417e5c12203915471c5006234ba1795a81407eb079acae1de1c914570e5d"} Oct 30 22:34:01.537458 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:01.537410 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" event={"ID":"772ecd37-9f05-48de-91e3-0ba670519f00","Type":"ContainerStarted","Data":"20a7715a7d12cf42257e7cf60160ed7c1709b739684198b6257439bade4338df"} Oct 30 22:34:01.649858 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:01.649779 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" podStartSLOduration=2.649761319 podStartE2EDuration="2.649761319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:01.649188431 +0000 UTC m=+2644.201364731" watchObservedRunningTime="2026-10-30 22:34:01.649761319 +0000 UTC m=+2644.201937620" Oct 30 22:34:01.742972 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:01.742932 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk"] Oct 30 22:34:01.749080 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:01.749048 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ed44da-5jfvk"] Oct 30 22:34:02.067514 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:02.067477 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" Oct 30 22:34:03.603533 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:03.603493 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd3ec6d4-0e2a-494d-908a-45080c2f9eb1" path="/var/lib/kubelet/pods/dd3ec6d4-0e2a-494d-908a-45080c2f9eb1/volumes" Oct 30 22:34:07.755234 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:07.755186 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p"] Oct 30 22:34:07.758689 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:07.758661 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-bc714d-kg28p"] Oct 30 22:34:09.604775 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:09.604738 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9083308-566b-4535-b9f0-630f6775bce0" path="/var/lib/kubelet/pods/e9083308-566b-4535-b9f0-630f6775bce0/volumes" Oct 30 22:34:09.868489 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:09.868396 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5"] Oct 30 22:34:09.868924 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:09.868880 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" podUID="aa2bdf13-4ade-4f01-8a3c-7463d62efd49" containerName="istio-proxy" containerID="cri-o://73d1fa5fe2df611adf86e78b3c65bfdb7a51fa8a1d40cc97a2be098d88e30093" gracePeriod=30 Oct 30 22:34:10.067328 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:10.067287 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" Oct 30 22:34:10.081174 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:10.081142 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" Oct 30 22:34:10.621147 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:10.621115 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" Oct 30 22:34:10.906120 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:10.906033 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz"] Oct 30 22:34:10.910715 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:10.910683 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz" Oct 30 22:34:10.936701 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:10.936662 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj6pz\" (UniqueName: \"kubernetes.io/projected/a05cb975-bd6e-484f-bcfb-8508885c7cb4-kube-api-access-pj6pz\") pod \"maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz\" (UID: \"a05cb975-bd6e-484f-bcfb-8508885c7cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz" Oct 30 22:34:10.936701 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:10.936704 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a05cb975-bd6e-484f-bcfb-8508885c7cb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz\" (UID: \"a05cb975-bd6e-484f-bcfb-8508885c7cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz" Oct 30 22:34:10.936982 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:10.936817 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a05cb975-bd6e-484f-bcfb-8508885c7cb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz\" (UID: \"a05cb975-bd6e-484f-bcfb-8508885c7cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz" Oct 30 22:34:10.962866 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:10.962833 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:34:10.963059 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:10.962881 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:34:10.964166 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:10.964142 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:34:10.964392 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:10.964375 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:34:10.965927 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:10.965907 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz"] Oct 30 22:34:11.038087 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:11.038046 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj6pz\" (UniqueName: \"kubernetes.io/projected/a05cb975-bd6e-484f-bcfb-8508885c7cb4-kube-api-access-pj6pz\") pod \"maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz\" (UID: \"a05cb975-bd6e-484f-bcfb-8508885c7cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz" Oct 30 22:34:11.038309 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:11.038100 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a05cb975-bd6e-484f-bcfb-8508885c7cb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz\" (UID: \"a05cb975-bd6e-484f-bcfb-8508885c7cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz" Oct 30 22:34:11.038309 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:11.038156 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a05cb975-bd6e-484f-bcfb-8508885c7cb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz\" (UID: \"a05cb975-bd6e-484f-bcfb-8508885c7cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz" Oct 30 22:34:11.038819 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:11.038797 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a05cb975-bd6e-484f-bcfb-8508885c7cb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz\" (UID: \"a05cb975-bd6e-484f-bcfb-8508885c7cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz" Oct 30 22:34:11.040669 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:11.040633 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a05cb975-bd6e-484f-bcfb-8508885c7cb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz\" (UID: \"a05cb975-bd6e-484f-bcfb-8508885c7cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz" Oct 30 22:34:11.097931 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:11.097901 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj6pz\" (UniqueName: \"kubernetes.io/projected/a05cb975-bd6e-484f-bcfb-8508885c7cb4-kube-api-access-pj6pz\") pod \"maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz\" (UID: \"a05cb975-bd6e-484f-bcfb-8508885c7cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz" Oct 30 22:34:11.225566 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:11.225477 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz" Oct 30 22:34:11.601745 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:34:11.601711 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05cb975_bd6e_484f_bcfb_8508885c7cb4.slice/crio-50f1d51d23f5c50061c48ade796ede7a65dc2192dc7c5c5a73694b0f6f2a17da WatchSource:0}: Error finding container 50f1d51d23f5c50061c48ade796ede7a65dc2192dc7c5c5a73694b0f6f2a17da: Status 404 returned error can't find the container with id 50f1d51d23f5c50061c48ade796ede7a65dc2192dc7c5c5a73694b0f6f2a17da Oct 30 22:34:11.605178 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:11.605153 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz"] Oct 30 22:34:11.621908 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:11.621868 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz" event={"ID":"a05cb975-bd6e-484f-bcfb-8508885c7cb4","Type":"ContainerStarted","Data":"50f1d51d23f5c50061c48ade796ede7a65dc2192dc7c5c5a73694b0f6f2a17da"} Oct 30 22:34:12.632805 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:12.632764 6006 generic.go:356] "Generic (PLEG): container finished" podID="a05cb975-bd6e-484f-bcfb-8508885c7cb4" containerID="431ad75dc4451a620bbbe589a7977f86df2eab14a3684253f5d3e237a87511df" exitCode=0 Oct 30 22:34:12.633224 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:12.632850 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz" event={"ID":"a05cb975-bd6e-484f-bcfb-8508885c7cb4","Type":"ContainerDied","Data":"431ad75dc4451a620bbbe589a7977f86df2eab14a3684253f5d3e237a87511df"} Oct 30 22:34:13.804504 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:13.804471 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz" Oct 30 22:34:13.872165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:13.872132 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a05cb975-bd6e-484f-bcfb-8508885c7cb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a05cb975-bd6e-484f-bcfb-8508885c7cb4-maas-api-xks-ca\") pod \"a05cb975-bd6e-484f-bcfb-8508885c7cb4\" (UID: \"a05cb975-bd6e-484f-bcfb-8508885c7cb4\") " Oct 30 22:34:13.872400 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:13.872305 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a05cb975-bd6e-484f-bcfb-8508885c7cb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a05cb975-bd6e-484f-bcfb-8508885c7cb4-maas-api-openshift-ca\") pod \"a05cb975-bd6e-484f-bcfb-8508885c7cb4\" (UID: \"a05cb975-bd6e-484f-bcfb-8508885c7cb4\") " Oct 30 22:34:13.872400 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:13.872369 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a05cb975-bd6e-484f-bcfb-8508885c7cb4-kube-api-access-pj6pz\" (UniqueName: \"kubernetes.io/projected/a05cb975-bd6e-484f-bcfb-8508885c7cb4-kube-api-access-pj6pz\") pod \"a05cb975-bd6e-484f-bcfb-8508885c7cb4\" (UID: \"a05cb975-bd6e-484f-bcfb-8508885c7cb4\") " Oct 30 22:34:13.872732 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:13.872698 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a05cb975-bd6e-484f-bcfb-8508885c7cb4-maas-api-openshift-ca" pod "a05cb975-bd6e-484f-bcfb-8508885c7cb4" (UID: "a05cb975-bd6e-484f-bcfb-8508885c7cb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:13.874342 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:13.874320 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a05cb975-bd6e-484f-bcfb-8508885c7cb4-maas-api-xks-ca" pod "a05cb975-bd6e-484f-bcfb-8508885c7cb4" (UID: "a05cb975-bd6e-484f-bcfb-8508885c7cb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:13.874565 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:13.874530 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a05cb975-bd6e-484f-bcfb-8508885c7cb4-kube-api-access-pj6pz" pod "a05cb975-bd6e-484f-bcfb-8508885c7cb4" (UID: "a05cb975-bd6e-484f-bcfb-8508885c7cb4"). InnerVolumeSpecName "kube-api-access-pj6pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:13.973406 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:13.973321 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a05cb975-bd6e-484f-bcfb-8508885c7cb4-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:13.973406 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:13.973352 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj6pz\" (UniqueName: \"kubernetes.io/projected/a05cb975-bd6e-484f-bcfb-8508885c7cb4-kube-api-access-pj6pz\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:13.973406 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:13.973363 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a05cb975-bd6e-484f-bcfb-8508885c7cb4-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:14.650950 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:14.650909 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz" event={"ID":"a05cb975-bd6e-484f-bcfb-8508885c7cb4","Type":"ContainerDied","Data":"50f1d51d23f5c50061c48ade796ede7a65dc2192dc7c5c5a73694b0f6f2a17da"} Oct 30 22:34:14.651146 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:14.650970 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50f1d51d23f5c50061c48ade796ede7a65dc2192dc7c5c5a73694b0f6f2a17da" Oct 30 22:34:14.651146 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:14.650932 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz" Oct 30 22:34:15.169186 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.169159 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:34:15.289302 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.289185 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-kube-api-access-nkh8k\" (UniqueName: \"kubernetes.io/projected/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-kube-api-access-nkh8k\") pod \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " Oct 30 22:34:15.289457 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.289334 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-workload-certs\") pod \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " Oct 30 22:34:15.289457 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.289385 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-data\") pod \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " Oct 30 22:34:15.289585 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.289462 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istiod-ca-cert\") pod \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " Oct 30 22:34:15.289585 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.289489 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-workload-socket\") pod \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " Oct 30 22:34:15.289585 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.289519 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-credential-socket\") pod \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " Oct 30 22:34:15.289585 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.289557 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-token\" (UniqueName: \"kubernetes.io/projected/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-token\") pod \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " Oct 30 22:34:15.289805 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.289619 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-envoy\") pod \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " Oct 30 22:34:15.289805 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.289664 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-podinfo\") pod \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\" (UID: \"aa2bdf13-4ade-4f01-8a3c-7463d62efd49\") " Oct 30 22:34:15.289805 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.289686 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-workload-certs" pod "aa2bdf13-4ade-4f01-8a3c-7463d62efd49" (UID: "aa2bdf13-4ade-4f01-8a3c-7463d62efd49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:15.289805 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.289754 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-data" pod "aa2bdf13-4ade-4f01-8a3c-7463d62efd49" (UID: "aa2bdf13-4ade-4f01-8a3c-7463d62efd49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:15.289805 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.289768 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-workload-socket" pod "aa2bdf13-4ade-4f01-8a3c-7463d62efd49" (UID: "aa2bdf13-4ade-4f01-8a3c-7463d62efd49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:15.290068 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.289941 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istiod-ca-cert" pod "aa2bdf13-4ade-4f01-8a3c-7463d62efd49" (UID: "aa2bdf13-4ade-4f01-8a3c-7463d62efd49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:15.290221 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.290190 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-credential-socket" pod "aa2bdf13-4ade-4f01-8a3c-7463d62efd49" (UID: "aa2bdf13-4ade-4f01-8a3c-7463d62efd49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:15.290404 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.290221 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:15.290404 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.290282 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:15.290404 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.290297 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:15.290404 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.290311 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:15.291904 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.291881 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-kube-api-access-nkh8k" pod "aa2bdf13-4ade-4f01-8a3c-7463d62efd49" (UID: "aa2bdf13-4ade-4f01-8a3c-7463d62efd49"). InnerVolumeSpecName "kube-api-access-nkh8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:15.292358 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.292334 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-podinfo" pod "aa2bdf13-4ade-4f01-8a3c-7463d62efd49" (UID: "aa2bdf13-4ade-4f01-8a3c-7463d62efd49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:34:15.292358 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.292354 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-envoy" pod "aa2bdf13-4ade-4f01-8a3c-7463d62efd49" (UID: "aa2bdf13-4ade-4f01-8a3c-7463d62efd49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:15.292536 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.292441 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-token" pod "aa2bdf13-4ade-4f01-8a3c-7463d62efd49" (UID: "aa2bdf13-4ade-4f01-8a3c-7463d62efd49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:15.391731 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.391696 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:15.391731 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.391729 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:15.391731 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.391739 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:15.391988 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.391747 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:15.391988 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.391757 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkh8k\" (UniqueName: \"kubernetes.io/projected/aa2bdf13-4ade-4f01-8a3c-7463d62efd49-kube-api-access-nkh8k\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:15.431859 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.431818 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp"] Oct 30 22:34:15.660550 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.660513 6006 generic.go:356] "Generic (PLEG): container finished" podID="aa2bdf13-4ade-4f01-8a3c-7463d62efd49" containerID="73d1fa5fe2df611adf86e78b3c65bfdb7a51fa8a1d40cc97a2be098d88e30093" exitCode=0 Oct 30 22:34:15.660772 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.660574 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" Oct 30 22:34:15.660772 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.660608 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" event={"ID":"aa2bdf13-4ade-4f01-8a3c-7463d62efd49","Type":"ContainerDied","Data":"73d1fa5fe2df611adf86e78b3c65bfdb7a51fa8a1d40cc97a2be098d88e30093"} Oct 30 22:34:15.660772 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.660676 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5" event={"ID":"aa2bdf13-4ade-4f01-8a3c-7463d62efd49","Type":"ContainerDied","Data":"90a1390c1964160424a539987111ff29a6be62557175fd4a01e15c87ee16273b"} Oct 30 22:34:15.660772 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.660706 6006 scope.go:122] "RemoveContainer" containerID="73d1fa5fe2df611adf86e78b3c65bfdb7a51fa8a1d40cc97a2be098d88e30093" Oct 30 22:34:15.673566 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.673540 6006 scope.go:122] "RemoveContainer" containerID="9e870379876ff473f4bf057ce3a7611f8d90365794d7086c38c4e32d4d5b3aa4" Oct 30 22:34:15.685123 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.685098 6006 scope.go:122] "RemoveContainer" containerID="73d1fa5fe2df611adf86e78b3c65bfdb7a51fa8a1d40cc97a2be098d88e30093" Oct 30 22:34:15.685511 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:34:15.685483 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d1fa5fe2df611adf86e78b3c65bfdb7a51fa8a1d40cc97a2be098d88e30093\": container with ID starting with 73d1fa5fe2df611adf86e78b3c65bfdb7a51fa8a1d40cc97a2be098d88e30093 not found: ID does not exist" containerID="73d1fa5fe2df611adf86e78b3c65bfdb7a51fa8a1d40cc97a2be098d88e30093" Oct 30 22:34:15.685640 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.685520 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d1fa5fe2df611adf86e78b3c65bfdb7a51fa8a1d40cc97a2be098d88e30093"} err="failed to get container status \"73d1fa5fe2df611adf86e78b3c65bfdb7a51fa8a1d40cc97a2be098d88e30093\": rpc error: code = NotFound desc = could not find container \"73d1fa5fe2df611adf86e78b3c65bfdb7a51fa8a1d40cc97a2be098d88e30093\": container with ID starting with 73d1fa5fe2df611adf86e78b3c65bfdb7a51fa8a1d40cc97a2be098d88e30093 not found: ID does not exist" Oct 30 22:34:15.685640 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.685542 6006 scope.go:122] "RemoveContainer" containerID="9e870379876ff473f4bf057ce3a7611f8d90365794d7086c38c4e32d4d5b3aa4" Oct 30 22:34:15.685826 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:34:15.685805 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e870379876ff473f4bf057ce3a7611f8d90365794d7086c38c4e32d4d5b3aa4\": container with ID starting with 9e870379876ff473f4bf057ce3a7611f8d90365794d7086c38c4e32d4d5b3aa4 not found: ID does not exist" containerID="9e870379876ff473f4bf057ce3a7611f8d90365794d7086c38c4e32d4d5b3aa4" Oct 30 22:34:15.685896 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.685834 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e870379876ff473f4bf057ce3a7611f8d90365794d7086c38c4e32d4d5b3aa4"} err="failed to get container status \"9e870379876ff473f4bf057ce3a7611f8d90365794d7086c38c4e32d4d5b3aa4\": rpc error: code = NotFound desc = could not find container \"9e870379876ff473f4bf057ce3a7611f8d90365794d7086c38c4e32d4d5b3aa4\": container with ID starting with 9e870379876ff473f4bf057ce3a7611f8d90365794d7086c38c4e32d4d5b3aa4 not found: ID does not exist" Oct 30 22:34:15.826973 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.826934 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5"] Oct 30 22:34:15.831630 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:15.831295 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ed44da-openshift-default-86bb476b97-jc5x5"] Oct 30 22:34:17.026111 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:17.026073 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:34:17.603117 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:17.603082 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa2bdf13-4ade-4f01-8a3c-7463d62efd49" path="/var/lib/kubelet/pods/aa2bdf13-4ade-4f01-8a3c-7463d62efd49/volumes" Oct 30 22:34:27.911144 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:34:27.911101 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:34:30.453240 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.453176 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" containerID="cri-o://53c7f3cd8d30cb1f2d26502be354bfef9fa843605a88e51f37b396a639512917" gracePeriod=45 Oct 30 22:34:30.718689 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.718622 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:34:30.766515 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.766476 6006 generic.go:356] "Generic (PLEG): container finished" podID="8040ac93-043b-49a2-8388-f44f1f93e440" containerID="53c7f3cd8d30cb1f2d26502be354bfef9fa843605a88e51f37b396a639512917" exitCode=0 Oct 30 22:34:30.766711 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.766533 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" Oct 30 22:34:30.766711 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.766565 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" event={"ID":"8040ac93-043b-49a2-8388-f44f1f93e440","Type":"ContainerDied","Data":"53c7f3cd8d30cb1f2d26502be354bfef9fa843605a88e51f37b396a639512917"} Oct 30 22:34:30.766711 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.766622 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp" event={"ID":"8040ac93-043b-49a2-8388-f44f1f93e440","Type":"ContainerDied","Data":"2b8918ed4e1590126c8aecc8ee84adb9a0e892be98769070369dbd061a62635f"} Oct 30 22:34:30.766711 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.766647 6006 scope.go:122] "RemoveContainer" containerID="53c7f3cd8d30cb1f2d26502be354bfef9fa843605a88e51f37b396a639512917" Oct 30 22:34:30.776447 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.776428 6006 scope.go:122] "RemoveContainer" containerID="53c7f3cd8d30cb1f2d26502be354bfef9fa843605a88e51f37b396a639512917" Oct 30 22:34:30.776771 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:34:30.776745 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53c7f3cd8d30cb1f2d26502be354bfef9fa843605a88e51f37b396a639512917\": container with ID starting with 53c7f3cd8d30cb1f2d26502be354bfef9fa843605a88e51f37b396a639512917 not found: ID does not exist" containerID="53c7f3cd8d30cb1f2d26502be354bfef9fa843605a88e51f37b396a639512917" Oct 30 22:34:30.776876 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.776781 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53c7f3cd8d30cb1f2d26502be354bfef9fa843605a88e51f37b396a639512917"} err="failed to get container status \"53c7f3cd8d30cb1f2d26502be354bfef9fa843605a88e51f37b396a639512917\": rpc error: code = NotFound desc = could not find container \"53c7f3cd8d30cb1f2d26502be354bfef9fa843605a88e51f37b396a639512917\": container with ID starting with 53c7f3cd8d30cb1f2d26502be354bfef9fa843605a88e51f37b396a639512917 not found: ID does not exist" Oct 30 22:34:30.876239 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.876201 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8040ac93-043b-49a2-8388-f44f1f93e440-kube-api-access-z8sjv\" (UniqueName: \"kubernetes.io/projected/8040ac93-043b-49a2-8388-f44f1f93e440-kube-api-access-z8sjv\") pod \"8040ac93-043b-49a2-8388-f44f1f93e440\" (UID: \"8040ac93-043b-49a2-8388-f44f1f93e440\") " Oct 30 22:34:30.876670 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.876643 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-dshm\") pod \"8040ac93-043b-49a2-8388-f44f1f93e440\" (UID: \"8040ac93-043b-49a2-8388-f44f1f93e440\") " Oct 30 22:34:30.876880 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.876859 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-home\" (UniqueName: \"kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-home\") pod \"8040ac93-043b-49a2-8388-f44f1f93e440\" (UID: \"8040ac93-043b-49a2-8388-f44f1f93e440\") " Oct 30 22:34:30.877030 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.877014 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-tmp-dir\") pod \"8040ac93-043b-49a2-8388-f44f1f93e440\" (UID: \"8040ac93-043b-49a2-8388-f44f1f93e440\") " Oct 30 22:34:30.877186 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.877169 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-model-cache\") pod \"8040ac93-043b-49a2-8388-f44f1f93e440\" (UID: \"8040ac93-043b-49a2-8388-f44f1f93e440\") " Oct 30 22:34:30.877346 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.877330 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8040ac93-043b-49a2-8388-f44f1f93e440-tls-certs\" (UniqueName: \"kubernetes.io/secret/8040ac93-043b-49a2-8388-f44f1f93e440-tls-certs\") pod \"8040ac93-043b-49a2-8388-f44f1f93e440\" (UID: \"8040ac93-043b-49a2-8388-f44f1f93e440\") " Oct 30 22:34:30.877529 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.877497 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-home" pod "8040ac93-043b-49a2-8388-f44f1f93e440" (UID: "8040ac93-043b-49a2-8388-f44f1f93e440"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:30.877906 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.877851 6006 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-home\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.878874 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.878401 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-tmp-dir" pod "8040ac93-043b-49a2-8388-f44f1f93e440" (UID: "8040ac93-043b-49a2-8388-f44f1f93e440"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:30.878874 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.878821 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-model-cache" pod "8040ac93-043b-49a2-8388-f44f1f93e440" (UID: "8040ac93-043b-49a2-8388-f44f1f93e440"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:30.884437 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.884405 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8040ac93-043b-49a2-8388-f44f1f93e440-kube-api-access-z8sjv" pod "8040ac93-043b-49a2-8388-f44f1f93e440" (UID: "8040ac93-043b-49a2-8388-f44f1f93e440"). InnerVolumeSpecName "kube-api-access-z8sjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:30.884875 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.884846 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-dshm" pod "8040ac93-043b-49a2-8388-f44f1f93e440" (UID: "8040ac93-043b-49a2-8388-f44f1f93e440"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:30.885050 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.885030 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8040ac93-043b-49a2-8388-f44f1f93e440-tls-certs" pod "8040ac93-043b-49a2-8388-f44f1f93e440" (UID: "8040ac93-043b-49a2-8388-f44f1f93e440"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:30.978686 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.978595 6006 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-tmp-dir\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.978686 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.978629 6006 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-model-cache\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.978686 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.978639 6006 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8040ac93-043b-49a2-8388-f44f1f93e440-tls-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.978686 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.978648 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8sjv\" (UniqueName: \"kubernetes.io/projected/8040ac93-043b-49a2-8388-f44f1f93e440-kube-api-access-z8sjv\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.978686 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:30.978658 6006 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8040ac93-043b-49a2-8388-f44f1f93e440-dshm\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:31.182345 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:31.182301 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp"] Oct 30 22:34:31.189186 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:31.189141 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-779f32/simulator-w0-779f32-kserve-85c494d775-d4bkp"] Oct 30 22:34:31.603379 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:31.603334 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" path="/var/lib/kubelet/pods/8040ac93-043b-49a2-8388-f44f1f93e440/volumes" Oct 30 22:34:37.219593 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:37.219546 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:34:37.407261 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:37.407202 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q"] Oct 30 22:34:37.407482 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:37.407450 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" podUID="b246ab27-ac36-46ef-9db0-93545e5baefb" containerName="istio-proxy" containerID="cri-o://e36996a5a1250e25b244d009a07c9367ec7910e9829bb232d852b502e8876595" gracePeriod=30 Oct 30 22:34:37.409458 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:37.409433 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8"] Oct 30 22:34:37.409631 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:37.409606 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" podUID="b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51" containerName="istio-proxy" containerID="cri-o://a43ea638943f8fbae09b1efced9f4ac0463316723cab5ad1f16e294e51833db7" gracePeriod=30 Oct 30 22:34:38.516968 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:38.516926 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-vnd7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:34:38.517404 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:38.517021 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" podUID="b246ab27-ac36-46ef-9db0-93545e5baefb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:34:42.688325 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.688241 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:34:42.709303 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.709276 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:34:42.709580 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.709565 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-credential-socket\") pod \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " Oct 30 22:34:42.709647 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.709621 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-workload-certs\") pod \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " Oct 30 22:34:42.709689 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.709676 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-envoy\") pod \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " Oct 30 22:34:42.709741 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.709727 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-token\" (UniqueName: \"kubernetes.io/projected/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-token\") pod \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " Oct 30 22:34:42.709849 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.709827 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-credential-socket" pod "b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51" (UID: "b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:42.709899 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.709851 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-podinfo\") pod \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " Oct 30 22:34:42.709899 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.709888 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-workload-socket\") pod \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " Oct 30 22:34:42.709984 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.709927 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-kube-api-access-bnm5x\" (UniqueName: \"kubernetes.io/projected/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-kube-api-access-bnm5x\") pod \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " Oct 30 22:34:42.710051 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.709979 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-data\") pod \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " Oct 30 22:34:42.710051 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.709927 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-workload-certs" pod "b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51" (UID: "b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:42.710051 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.710030 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istiod-ca-cert\") pod \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\" (UID: \"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51\") " Oct 30 22:34:42.710217 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.710137 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-workload-socket" pod "b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51" (UID: "b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:42.710305 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.710279 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-data" pod "b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51" (UID: "b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:42.710497 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.710469 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istiod-ca-cert" pod "b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51" (UID: "b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:42.710661 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.710642 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:42.710736 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.710669 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:42.710736 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.710683 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:42.710736 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.710698 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:42.710736 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.710713 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:42.712325 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.712294 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-kube-api-access-bnm5x" pod "b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51" (UID: "b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51"). InnerVolumeSpecName "kube-api-access-bnm5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:42.712437 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.712360 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-token" pod "b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51" (UID: "b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:42.712529 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.712515 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-podinfo" pod "b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51" (UID: "b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:34:42.712685 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.712662 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-envoy" pod "b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51" (UID: "b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:42.811309 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.811245 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-token\" (UniqueName: \"kubernetes.io/projected/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-token\") pod \"b246ab27-ac36-46ef-9db0-93545e5baefb\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " Oct 30 22:34:42.811309 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.811317 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b246ab27-ac36-46ef-9db0-93545e5baefb-kube-api-access-tzr8n\" (UniqueName: \"kubernetes.io/projected/b246ab27-ac36-46ef-9db0-93545e5baefb-kube-api-access-tzr8n\") pod \"b246ab27-ac36-46ef-9db0-93545e5baefb\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " Oct 30 22:34:42.811538 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.811348 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-workload-socket\") pod \"b246ab27-ac36-46ef-9db0-93545e5baefb\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " Oct 30 22:34:42.811538 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.811386 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-workload-certs\") pod \"b246ab27-ac36-46ef-9db0-93545e5baefb\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " Oct 30 22:34:42.811538 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.811429 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-envoy\") pod \"b246ab27-ac36-46ef-9db0-93545e5baefb\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " Oct 30 22:34:42.811538 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.811481 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b246ab27-ac36-46ef-9db0-93545e5baefb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b246ab27-ac36-46ef-9db0-93545e5baefb-istiod-ca-cert\") pod \"b246ab27-ac36-46ef-9db0-93545e5baefb\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " Oct 30 22:34:42.811538 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.811500 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-credential-socket\") pod \"b246ab27-ac36-46ef-9db0-93545e5baefb\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " Oct 30 22:34:42.811538 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.811534 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-podinfo\") pod \"b246ab27-ac36-46ef-9db0-93545e5baefb\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " Oct 30 22:34:42.811749 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.811622 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-data\") pod \"b246ab27-ac36-46ef-9db0-93545e5baefb\" (UID: \"b246ab27-ac36-46ef-9db0-93545e5baefb\") " Oct 30 22:34:42.811875 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.811841 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:42.811875 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.811864 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:42.811875 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.811880 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnm5x\" (UniqueName: \"kubernetes.io/projected/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-kube-api-access-bnm5x\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:42.812128 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.811894 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:42.812198 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.812114 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-workload-certs" pod "b246ab27-ac36-46ef-9db0-93545e5baefb" (UID: "b246ab27-ac36-46ef-9db0-93545e5baefb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:42.812289 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.812208 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-credential-socket" pod "b246ab27-ac36-46ef-9db0-93545e5baefb" (UID: "b246ab27-ac36-46ef-9db0-93545e5baefb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:42.812345 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.812321 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-workload-socket" pod "b246ab27-ac36-46ef-9db0-93545e5baefb" (UID: "b246ab27-ac36-46ef-9db0-93545e5baefb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:42.812410 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.812355 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b246ab27-ac36-46ef-9db0-93545e5baefb-istiod-ca-cert" pod "b246ab27-ac36-46ef-9db0-93545e5baefb" (UID: "b246ab27-ac36-46ef-9db0-93545e5baefb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:42.812552 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.812531 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-data" pod "b246ab27-ac36-46ef-9db0-93545e5baefb" (UID: "b246ab27-ac36-46ef-9db0-93545e5baefb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:42.814292 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.814222 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-envoy" pod "b246ab27-ac36-46ef-9db0-93545e5baefb" (UID: "b246ab27-ac36-46ef-9db0-93545e5baefb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:42.814387 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.814293 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-token" pod "b246ab27-ac36-46ef-9db0-93545e5baefb" (UID: "b246ab27-ac36-46ef-9db0-93545e5baefb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:42.814545 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.814528 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-podinfo" pod "b246ab27-ac36-46ef-9db0-93545e5baefb" (UID: "b246ab27-ac36-46ef-9db0-93545e5baefb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:34:42.814608 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.814559 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b246ab27-ac36-46ef-9db0-93545e5baefb-kube-api-access-tzr8n" pod "b246ab27-ac36-46ef-9db0-93545e5baefb" (UID: "b246ab27-ac36-46ef-9db0-93545e5baefb"). InnerVolumeSpecName "kube-api-access-tzr8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:42.849140 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.849102 6006 generic.go:356] "Generic (PLEG): container finished" podID="b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51" containerID="a43ea638943f8fbae09b1efced9f4ac0463316723cab5ad1f16e294e51833db7" exitCode=0 Oct 30 22:34:42.849382 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.849165 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" Oct 30 22:34:42.849382 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.849202 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" event={"ID":"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51","Type":"ContainerDied","Data":"a43ea638943f8fbae09b1efced9f4ac0463316723cab5ad1f16e294e51833db7"} Oct 30 22:34:42.849510 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.849481 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8" event={"ID":"b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51","Type":"ContainerDied","Data":"4048914dda616008a4ff94dea99ad8da7b9dff94df226f31a1892f9ec5e2150a"} Oct 30 22:34:42.849605 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.849538 6006 scope.go:122] "RemoveContainer" containerID="a43ea638943f8fbae09b1efced9f4ac0463316723cab5ad1f16e294e51833db7" Oct 30 22:34:42.852686 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.852171 6006 generic.go:356] "Generic (PLEG): container finished" podID="b246ab27-ac36-46ef-9db0-93545e5baefb" containerID="e36996a5a1250e25b244d009a07c9367ec7910e9829bb232d852b502e8876595" exitCode=0 Oct 30 22:34:42.852686 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.852393 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" event={"ID":"b246ab27-ac36-46ef-9db0-93545e5baefb","Type":"ContainerDied","Data":"e36996a5a1250e25b244d009a07c9367ec7910e9829bb232d852b502e8876595"} Oct 30 22:34:42.852686 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.852445 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" event={"ID":"b246ab27-ac36-46ef-9db0-93545e5baefb","Type":"ContainerDied","Data":"0648ca28834cb9d8c62175798d839d04bc55b918c02e1ffb1b39c12899479e57"} Oct 30 22:34:42.852686 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.852518 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q" Oct 30 22:34:42.864163 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.864138 6006 scope.go:122] "RemoveContainer" containerID="a43ea638943f8fbae09b1efced9f4ac0463316723cab5ad1f16e294e51833db7" Oct 30 22:34:42.864619 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:34:42.864516 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a43ea638943f8fbae09b1efced9f4ac0463316723cab5ad1f16e294e51833db7\": container with ID starting with a43ea638943f8fbae09b1efced9f4ac0463316723cab5ad1f16e294e51833db7 not found: ID does not exist" containerID="a43ea638943f8fbae09b1efced9f4ac0463316723cab5ad1f16e294e51833db7" Oct 30 22:34:42.864619 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.864551 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a43ea638943f8fbae09b1efced9f4ac0463316723cab5ad1f16e294e51833db7"} err="failed to get container status \"a43ea638943f8fbae09b1efced9f4ac0463316723cab5ad1f16e294e51833db7\": rpc error: code = NotFound desc = could not find container \"a43ea638943f8fbae09b1efced9f4ac0463316723cab5ad1f16e294e51833db7\": container with ID starting with a43ea638943f8fbae09b1efced9f4ac0463316723cab5ad1f16e294e51833db7 not found: ID does not exist" Oct 30 22:34:42.864619 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.864575 6006 scope.go:122] "RemoveContainer" containerID="e36996a5a1250e25b244d009a07c9367ec7910e9829bb232d852b502e8876595" Oct 30 22:34:42.877876 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.877852 6006 scope.go:122] "RemoveContainer" containerID="e36996a5a1250e25b244d009a07c9367ec7910e9829bb232d852b502e8876595" Oct 30 22:34:42.878240 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:34:42.878213 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36996a5a1250e25b244d009a07c9367ec7910e9829bb232d852b502e8876595\": container with ID starting with e36996a5a1250e25b244d009a07c9367ec7910e9829bb232d852b502e8876595 not found: ID does not exist" containerID="e36996a5a1250e25b244d009a07c9367ec7910e9829bb232d852b502e8876595" Oct 30 22:34:42.878378 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.878335 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36996a5a1250e25b244d009a07c9367ec7910e9829bb232d852b502e8876595"} err="failed to get container status \"e36996a5a1250e25b244d009a07c9367ec7910e9829bb232d852b502e8876595\": rpc error: code = NotFound desc = could not find container \"e36996a5a1250e25b244d009a07c9367ec7910e9829bb232d852b502e8876595\": container with ID starting with e36996a5a1250e25b244d009a07c9367ec7910e9829bb232d852b502e8876595 not found: ID does not exist" Oct 30 22:34:42.912534 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.912501 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b246ab27-ac36-46ef-9db0-93545e5baefb-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:42.912534 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.912535 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:42.912742 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.912547 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:42.912742 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.912558 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:42.912742 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.912567 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:42.912742 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.912576 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzr8n\" (UniqueName: \"kubernetes.io/projected/b246ab27-ac36-46ef-9db0-93545e5baefb-kube-api-access-tzr8n\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:42.912742 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.912585 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:42.912742 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.912594 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:42.912742 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:42.912603 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b246ab27-ac36-46ef-9db0-93545e5baefb-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.118921 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:43.118882 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q"] Oct 30 22:34:43.126359 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:43.126316 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vnd7q"] Oct 30 22:34:43.241915 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:43.241862 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8"] Oct 30 22:34:43.244666 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:43.244622 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sn4t8"] Oct 30 22:34:43.607000 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:43.606959 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b246ab27-ac36-46ef-9db0-93545e5baefb" path="/var/lib/kubelet/pods/b246ab27-ac36-46ef-9db0-93545e5baefb/volumes" Oct 30 22:34:43.607506 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:43.607487 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51" path="/var/lib/kubelet/pods/b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51/volumes" Oct 30 22:34:57.912352 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:34:57.912307 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:34:58.059167 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.059127 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1c51b57-kf674"] Oct 30 22:34:58.101620 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.101589 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 22:34:58.106683 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.106649 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e6f692b-wb4ls"] Oct 30 22:34:58.108474 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.108446 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1c51b57-kf674"] Oct 30 22:34:58.111183 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.111149 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e6f692b-wb4ls"] Oct 30 22:34:58.120662 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.120640 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 22:34:58.856001 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.855966 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b597e2f3-qbxfb"] Oct 30 22:34:58.856444 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.856429 6006 cpu_manager.go:414] "containerMap: removing container" podUID="a05cb975-bd6e-484f-bcfb-8508885c7cb4" containerName="revoke-keys" Oct 30 22:34:58.856512 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.856444 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a05cb975-bd6e-484f-bcfb-8508885c7cb4" containerName="revoke-keys" Oct 30 22:34:58.856512 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.856463 6006 cpu_manager.go:414] "containerMap: removing container" podUID="aa2bdf13-4ade-4f01-8a3c-7463d62efd49" containerName="istio-proxy" Oct 30 22:34:58.856512 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.856468 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa2bdf13-4ade-4f01-8a3c-7463d62efd49" containerName="istio-proxy" Oct 30 22:34:58.856512 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.856493 6006 cpu_manager.go:414] "containerMap: removing container" podUID="b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51" containerName="istio-proxy" Oct 30 22:34:58.856512 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.856498 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51" containerName="istio-proxy" Oct 30 22:34:58.856512 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.856506 6006 cpu_manager.go:414] "containerMap: removing container" podUID="b246ab27-ac36-46ef-9db0-93545e5baefb" containerName="istio-proxy" Oct 30 22:34:58.856512 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.856512 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b246ab27-ac36-46ef-9db0-93545e5baefb" containerName="istio-proxy" Oct 30 22:34:58.856720 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.856525 6006 cpu_manager.go:414] "containerMap: removing container" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" Oct 30 22:34:58.856720 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.856530 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" Oct 30 22:34:58.856720 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.856592 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4ae69d7-186a-4ff2-bde0-e95e0bc5bb51" containerName="istio-proxy" Oct 30 22:34:58.856720 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.856599 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa2bdf13-4ade-4f01-8a3c-7463d62efd49" containerName="istio-proxy" Oct 30 22:34:58.856720 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.856605 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="b246ab27-ac36-46ef-9db0-93545e5baefb" containerName="istio-proxy" Oct 30 22:34:58.856720 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.856615 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="8040ac93-043b-49a2-8388-f44f1f93e440" containerName="main" Oct 30 22:34:58.856720 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.856623 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="a05cb975-bd6e-484f-bcfb-8508885c7cb4" containerName="revoke-keys" Oct 30 22:34:58.859961 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.859945 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b597e2f3-qbxfb" Oct 30 22:34:58.916289 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.916244 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee281591-d6ztp"] Oct 30 22:34:58.916731 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.916718 6006 cpu_manager.go:414] "containerMap: removing container" podUID="aa2bdf13-4ade-4f01-8a3c-7463d62efd49" containerName="istio-proxy" Oct 30 22:34:58.916792 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.916732 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa2bdf13-4ade-4f01-8a3c-7463d62efd49" containerName="istio-proxy" Oct 30 22:34:58.916848 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.916839 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa2bdf13-4ade-4f01-8a3c-7463d62efd49" containerName="istio-proxy" Oct 30 22:34:58.920554 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.920525 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee281591-d6ztp" Oct 30 22:34:58.959488 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.959453 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:34:58.959488 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.959476 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:34:58.959765 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.959507 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:34:58.959765 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.959566 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:34:58.962608 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.962586 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b597e2f3-qbxfb"] Oct 30 22:34:58.977663 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.977628 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee281591-d6ztp"] Oct 30 22:34:58.988002 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.987972 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqvkl\" (UniqueName: \"kubernetes.io/projected/2011ac7e-5d4d-4368-8a5a-55cdd5de789e-kube-api-access-kqvkl\") pod \"maas-api-revoke-sub-ee281591-d6ztp\" (UID: \"2011ac7e-5d4d-4368-8a5a-55cdd5de789e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee281591-d6ztp" Oct 30 22:34:58.988167 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.988139 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4660bcf0-9e67-41c3-b480-c6b25dfbc9b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b597e2f3-qbxfb\" (UID: \"4660bcf0-9e67-41c3-b480-c6b25dfbc9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b597e2f3-qbxfb" Oct 30 22:34:58.988281 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.988223 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m96g\" (UniqueName: \"kubernetes.io/projected/4660bcf0-9e67-41c3-b480-c6b25dfbc9b9-kube-api-access-9m96g\") pod \"maas-api-revoke-sub-b597e2f3-qbxfb\" (UID: \"4660bcf0-9e67-41c3-b480-c6b25dfbc9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b597e2f3-qbxfb" Oct 30 22:34:58.988351 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.988317 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2011ac7e-5d4d-4368-8a5a-55cdd5de789e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee281591-d6ztp\" (UID: \"2011ac7e-5d4d-4368-8a5a-55cdd5de789e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee281591-d6ztp" Oct 30 22:34:58.988419 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.988396 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2011ac7e-5d4d-4368-8a5a-55cdd5de789e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee281591-d6ztp\" (UID: \"2011ac7e-5d4d-4368-8a5a-55cdd5de789e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee281591-d6ztp" Oct 30 22:34:58.988533 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:58.988508 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4660bcf0-9e67-41c3-b480-c6b25dfbc9b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b597e2f3-qbxfb\" (UID: \"4660bcf0-9e67-41c3-b480-c6b25dfbc9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b597e2f3-qbxfb" Oct 30 22:34:59.089920 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.089880 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2011ac7e-5d4d-4368-8a5a-55cdd5de789e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee281591-d6ztp\" (UID: \"2011ac7e-5d4d-4368-8a5a-55cdd5de789e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee281591-d6ztp" Oct 30 22:34:59.090126 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.089932 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4660bcf0-9e67-41c3-b480-c6b25dfbc9b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b597e2f3-qbxfb\" (UID: \"4660bcf0-9e67-41c3-b480-c6b25dfbc9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b597e2f3-qbxfb" Oct 30 22:34:59.090126 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.089965 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqvkl\" (UniqueName: \"kubernetes.io/projected/2011ac7e-5d4d-4368-8a5a-55cdd5de789e-kube-api-access-kqvkl\") pod \"maas-api-revoke-sub-ee281591-d6ztp\" (UID: \"2011ac7e-5d4d-4368-8a5a-55cdd5de789e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee281591-d6ztp" Oct 30 22:34:59.090126 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.090027 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4660bcf0-9e67-41c3-b480-c6b25dfbc9b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b597e2f3-qbxfb\" (UID: \"4660bcf0-9e67-41c3-b480-c6b25dfbc9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b597e2f3-qbxfb" Oct 30 22:34:59.090126 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.090052 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m96g\" (UniqueName: \"kubernetes.io/projected/4660bcf0-9e67-41c3-b480-c6b25dfbc9b9-kube-api-access-9m96g\") pod \"maas-api-revoke-sub-b597e2f3-qbxfb\" (UID: \"4660bcf0-9e67-41c3-b480-c6b25dfbc9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b597e2f3-qbxfb" Oct 30 22:34:59.090126 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.090093 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2011ac7e-5d4d-4368-8a5a-55cdd5de789e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee281591-d6ztp\" (UID: \"2011ac7e-5d4d-4368-8a5a-55cdd5de789e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee281591-d6ztp" Oct 30 22:34:59.090890 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.090867 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4660bcf0-9e67-41c3-b480-c6b25dfbc9b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b597e2f3-qbxfb\" (UID: \"4660bcf0-9e67-41c3-b480-c6b25dfbc9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b597e2f3-qbxfb" Oct 30 22:34:59.091009 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.090875 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2011ac7e-5d4d-4368-8a5a-55cdd5de789e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee281591-d6ztp\" (UID: \"2011ac7e-5d4d-4368-8a5a-55cdd5de789e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee281591-d6ztp" Oct 30 22:34:59.093013 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.092988 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4660bcf0-9e67-41c3-b480-c6b25dfbc9b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b597e2f3-qbxfb\" (UID: \"4660bcf0-9e67-41c3-b480-c6b25dfbc9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b597e2f3-qbxfb" Oct 30 22:34:59.093107 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.092995 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2011ac7e-5d4d-4368-8a5a-55cdd5de789e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee281591-d6ztp\" (UID: \"2011ac7e-5d4d-4368-8a5a-55cdd5de789e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee281591-d6ztp" Oct 30 22:34:59.149002 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.148917 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m96g\" (UniqueName: \"kubernetes.io/projected/4660bcf0-9e67-41c3-b480-c6b25dfbc9b9-kube-api-access-9m96g\") pod \"maas-api-revoke-sub-b597e2f3-qbxfb\" (UID: \"4660bcf0-9e67-41c3-b480-c6b25dfbc9b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b597e2f3-qbxfb" Oct 30 22:34:59.149476 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.149451 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqvkl\" (UniqueName: \"kubernetes.io/projected/2011ac7e-5d4d-4368-8a5a-55cdd5de789e-kube-api-access-kqvkl\") pod \"maas-api-revoke-sub-ee281591-d6ztp\" (UID: \"2011ac7e-5d4d-4368-8a5a-55cdd5de789e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee281591-d6ztp" Oct 30 22:34:59.173596 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.173563 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b597e2f3-qbxfb" Oct 30 22:34:59.235908 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.235874 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee281591-d6ztp" Oct 30 22:34:59.396314 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.396242 6006 scope.go:122] "RemoveContainer" containerID="7e7c601a5291b50a4df14a810e987247fb56aad07b733680353e806077b931ff" Oct 30 22:34:59.409819 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.409790 6006 scope.go:122] "RemoveContainer" containerID="935ed079f2936214634d3659d7a6e4361fc3f04d0ec53ffa7bd410e62dea7dee" Oct 30 22:34:59.421648 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.421618 6006 scope.go:122] "RemoveContainer" containerID="bc943006670f01789c0d71f38305737d128cf76e94c18b958a990e99c24cf7bc" Oct 30 22:34:59.432591 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.432563 6006 scope.go:122] "RemoveContainer" containerID="9ae75b13eb448049ab4337698d182de7d74239ecf4324004293d6b07747c5be8" Oct 30 22:34:59.443221 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.443198 6006 scope.go:122] "RemoveContainer" containerID="43549652394bbc3ee6bcbae1240962e1de7db88adb665819a4556ea9ccb0e512" Oct 30 22:34:59.452817 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.452794 6006 scope.go:122] "RemoveContainer" containerID="ab25ce38de2e12c84f7fc71e76f106f4b1d462963bef1a047ce1dc7217dddf0b" Oct 30 22:34:59.545582 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.545550 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b597e2f3-qbxfb"] Oct 30 22:34:59.549234 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:34:59.549200 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4660bcf0_9e67_41c3_b480_c6b25dfbc9b9.slice/crio-8a22d9aaff116456a4c8874ca8f9a4626a0026d6d6af907dd999a7dfbceb633b WatchSource:0}: Error finding container 8a22d9aaff116456a4c8874ca8f9a4626a0026d6d6af907dd999a7dfbceb633b: Status 404 returned error can't find the container with id 8a22d9aaff116456a4c8874ca8f9a4626a0026d6d6af907dd999a7dfbceb633b Oct 30 22:34:59.610727 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.610688 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="400871ab-5b3b-4b9c-8893-75e692a1e551" path="/var/lib/kubelet/pods/400871ab-5b3b-4b9c-8893-75e692a1e551/volumes" Oct 30 22:34:59.611275 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.611226 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dac4ba6-533e-4bba-89ed-5679379d5f62" path="/var/lib/kubelet/pods/8dac4ba6-533e-4bba-89ed-5679379d5f62/volumes" Oct 30 22:34:59.658338 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.658310 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee281591-d6ztp"] Oct 30 22:34:59.660459 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:34:59.660359 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2011ac7e_5d4d_4368_8a5a_55cdd5de789e.slice/crio-c2c41e8b56f84ba6705705a63a671d2dfca875846e52eadca5b84acd7466d85b WatchSource:0}: Error finding container c2c41e8b56f84ba6705705a63a671d2dfca875846e52eadca5b84acd7466d85b: Status 404 returned error can't find the container with id c2c41e8b56f84ba6705705a63a671d2dfca875846e52eadca5b84acd7466d85b Oct 30 22:34:59.984594 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.984494 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b597e2f3-qbxfb" event={"ID":"4660bcf0-9e67-41c3-b480-c6b25dfbc9b9","Type":"ContainerStarted","Data":"0e8a83f72f41e90bbffb2130329ad438df892305fc0fda376d9d101c47973a3b"} Oct 30 22:34:59.984594 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.984562 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b597e2f3-qbxfb" event={"ID":"4660bcf0-9e67-41c3-b480-c6b25dfbc9b9","Type":"ContainerStarted","Data":"8a22d9aaff116456a4c8874ca8f9a4626a0026d6d6af907dd999a7dfbceb633b"} Oct 30 22:34:59.985858 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:34:59.985834 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee281591-d6ztp" event={"ID":"2011ac7e-5d4d-4368-8a5a-55cdd5de789e","Type":"ContainerStarted","Data":"c2c41e8b56f84ba6705705a63a671d2dfca875846e52eadca5b84acd7466d85b"} Oct 30 22:35:00.095757 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:00.095697 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b597e2f3-qbxfb" podStartSLOduration=2.095682931 podStartE2EDuration="2.095682931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:00.093554109 +0000 UTC m=+2702.645730409" watchObservedRunningTime="2026-10-30 22:35:00.095682931 +0000 UTC m=+2702.647859230" Oct 30 22:35:00.993057 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:00.993019 6006 generic.go:356] "Generic (PLEG): container finished" podID="2011ac7e-5d4d-4368-8a5a-55cdd5de789e" containerID="26ab1cd0d6d6f18b5340687e0ba32c70cbae8c07ef82d26c77d159ab09331c83" exitCode=0 Oct 30 22:35:00.993466 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:00.993104 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee281591-d6ztp" event={"ID":"2011ac7e-5d4d-4368-8a5a-55cdd5de789e","Type":"ContainerDied","Data":"26ab1cd0d6d6f18b5340687e0ba32c70cbae8c07ef82d26c77d159ab09331c83"} Oct 30 22:35:00.994709 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:00.994683 6006 generic.go:356] "Generic (PLEG): container finished" podID="4660bcf0-9e67-41c3-b480-c6b25dfbc9b9" containerID="0e8a83f72f41e90bbffb2130329ad438df892305fc0fda376d9d101c47973a3b" exitCode=0 Oct 30 22:35:00.994824 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:00.994758 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b597e2f3-qbxfb" event={"ID":"4660bcf0-9e67-41c3-b480-c6b25dfbc9b9","Type":"ContainerDied","Data":"0e8a83f72f41e90bbffb2130329ad438df892305fc0fda376d9d101c47973a3b"} Oct 30 22:35:02.188492 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:02.188464 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b597e2f3-qbxfb" Oct 30 22:35:02.191752 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:02.191733 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee281591-d6ztp" Oct 30 22:35:02.345674 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:02.345638 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2011ac7e-5d4d-4368-8a5a-55cdd5de789e-kube-api-access-kqvkl\" (UniqueName: \"kubernetes.io/projected/2011ac7e-5d4d-4368-8a5a-55cdd5de789e-kube-api-access-kqvkl\") pod \"2011ac7e-5d4d-4368-8a5a-55cdd5de789e\" (UID: \"2011ac7e-5d4d-4368-8a5a-55cdd5de789e\") " Oct 30 22:35:02.345885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:02.345701 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4660bcf0-9e67-41c3-b480-c6b25dfbc9b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4660bcf0-9e67-41c3-b480-c6b25dfbc9b9-maas-api-openshift-ca\") pod \"4660bcf0-9e67-41c3-b480-c6b25dfbc9b9\" (UID: \"4660bcf0-9e67-41c3-b480-c6b25dfbc9b9\") " Oct 30 22:35:02.345885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:02.345725 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2011ac7e-5d4d-4368-8a5a-55cdd5de789e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2011ac7e-5d4d-4368-8a5a-55cdd5de789e-maas-api-openshift-ca\") pod \"2011ac7e-5d4d-4368-8a5a-55cdd5de789e\" (UID: \"2011ac7e-5d4d-4368-8a5a-55cdd5de789e\") " Oct 30 22:35:02.345885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:02.345770 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2011ac7e-5d4d-4368-8a5a-55cdd5de789e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2011ac7e-5d4d-4368-8a5a-55cdd5de789e-maas-api-xks-ca\") pod \"2011ac7e-5d4d-4368-8a5a-55cdd5de789e\" (UID: \"2011ac7e-5d4d-4368-8a5a-55cdd5de789e\") " Oct 30 22:35:02.345885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:02.345844 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4660bcf0-9e67-41c3-b480-c6b25dfbc9b9-kube-api-access-9m96g\" (UniqueName: \"kubernetes.io/projected/4660bcf0-9e67-41c3-b480-c6b25dfbc9b9-kube-api-access-9m96g\") pod \"4660bcf0-9e67-41c3-b480-c6b25dfbc9b9\" (UID: \"4660bcf0-9e67-41c3-b480-c6b25dfbc9b9\") " Oct 30 22:35:02.345885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:02.345883 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4660bcf0-9e67-41c3-b480-c6b25dfbc9b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4660bcf0-9e67-41c3-b480-c6b25dfbc9b9-maas-api-xks-ca\") pod \"4660bcf0-9e67-41c3-b480-c6b25dfbc9b9\" (UID: \"4660bcf0-9e67-41c3-b480-c6b25dfbc9b9\") " Oct 30 22:35:02.346524 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:02.346146 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2011ac7e-5d4d-4368-8a5a-55cdd5de789e-maas-api-openshift-ca" pod "2011ac7e-5d4d-4368-8a5a-55cdd5de789e" (UID: "2011ac7e-5d4d-4368-8a5a-55cdd5de789e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:02.346524 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:02.346340 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2011ac7e-5d4d-4368-8a5a-55cdd5de789e-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:02.346599 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:02.346572 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4660bcf0-9e67-41c3-b480-c6b25dfbc9b9-maas-api-openshift-ca" pod "4660bcf0-9e67-41c3-b480-c6b25dfbc9b9" (UID: "4660bcf0-9e67-41c3-b480-c6b25dfbc9b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:02.352413 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:02.350137 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2011ac7e-5d4d-4368-8a5a-55cdd5de789e-kube-api-access-kqvkl" pod "2011ac7e-5d4d-4368-8a5a-55cdd5de789e" (UID: "2011ac7e-5d4d-4368-8a5a-55cdd5de789e"). InnerVolumeSpecName "kube-api-access-kqvkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:02.352413 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:02.351060 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2011ac7e-5d4d-4368-8a5a-55cdd5de789e-maas-api-xks-ca" pod "2011ac7e-5d4d-4368-8a5a-55cdd5de789e" (UID: "2011ac7e-5d4d-4368-8a5a-55cdd5de789e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:02.352413 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:02.351086 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4660bcf0-9e67-41c3-b480-c6b25dfbc9b9-kube-api-access-9m96g" pod "4660bcf0-9e67-41c3-b480-c6b25dfbc9b9" (UID: "4660bcf0-9e67-41c3-b480-c6b25dfbc9b9"). InnerVolumeSpecName "kube-api-access-9m96g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:02.352413 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:02.351418 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4660bcf0-9e67-41c3-b480-c6b25dfbc9b9-maas-api-xks-ca" pod "4660bcf0-9e67-41c3-b480-c6b25dfbc9b9" (UID: "4660bcf0-9e67-41c3-b480-c6b25dfbc9b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:02.447898 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:02.447852 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqvkl\" (UniqueName: \"kubernetes.io/projected/2011ac7e-5d4d-4368-8a5a-55cdd5de789e-kube-api-access-kqvkl\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:02.447898 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:02.447898 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4660bcf0-9e67-41c3-b480-c6b25dfbc9b9-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:02.448133 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:02.447917 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2011ac7e-5d4d-4368-8a5a-55cdd5de789e-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:02.448133 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:02.447933 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m96g\" (UniqueName: \"kubernetes.io/projected/4660bcf0-9e67-41c3-b480-c6b25dfbc9b9-kube-api-access-9m96g\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:02.448133 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:02.447947 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4660bcf0-9e67-41c3-b480-c6b25dfbc9b9-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:02.586310 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:02.586267 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:35:03.009948 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:03.009912 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee281591-d6ztp" Oct 30 22:35:03.010174 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:03.009945 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee281591-d6ztp" event={"ID":"2011ac7e-5d4d-4368-8a5a-55cdd5de789e","Type":"ContainerDied","Data":"c2c41e8b56f84ba6705705a63a671d2dfca875846e52eadca5b84acd7466d85b"} Oct 30 22:35:03.010174 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:03.010004 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2c41e8b56f84ba6705705a63a671d2dfca875846e52eadca5b84acd7466d85b" Oct 30 22:35:03.011760 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:03.011720 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b597e2f3-qbxfb" event={"ID":"4660bcf0-9e67-41c3-b480-c6b25dfbc9b9","Type":"ContainerDied","Data":"8a22d9aaff116456a4c8874ca8f9a4626a0026d6d6af907dd999a7dfbceb633b"} Oct 30 22:35:03.011926 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:03.011771 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a22d9aaff116456a4c8874ca8f9a4626a0026d6d6af907dd999a7dfbceb633b" Oct 30 22:35:03.011926 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:03.011730 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b597e2f3-qbxfb" Oct 30 22:35:27.911148 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:35:27.911056 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:35:47.339887 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:47.339850 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz"] Oct 30 22:35:47.386381 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:47.386345 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-779f32-nrvpz"] Oct 30 22:35:47.610397 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:47.610300 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a05cb975-bd6e-484f-bcfb-8508885c7cb4" path="/var/lib/kubelet/pods/a05cb975-bd6e-484f-bcfb-8508885c7cb4/volumes" Oct 30 22:35:57.152663 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:57.152621 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:35:57.952648 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:35:57.952608 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:35:59.516679 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:59.516649 6006 scope.go:122] "RemoveContainer" containerID="95043e8d8433543ad8098d67d9e79837bbdddd960c68ba90a8ae94ecb309224f" Oct 30 22:35:59.526496 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:59.526472 6006 scope.go:122] "RemoveContainer" containerID="18917dd94687da1776b0a4814e7e6205e434862e9332a8d132d1f58bf199b7ee" Oct 30 22:35:59.535352 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:59.535332 6006 scope.go:122] "RemoveContainer" containerID="b7f7e44e09945fd8e5ad2160a0804761ebd263c00b1c4790fff25c77cc19d06b" Oct 30 22:35:59.546415 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:59.546389 6006 scope.go:122] "RemoveContainer" containerID="72d4e659cac1f60b4f84efc1c361bca3f42b34427a6e48817eae709b21315c72" Oct 30 22:35:59.557157 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:35:59.557132 6006 scope.go:122] "RemoveContainer" containerID="776d84e7a34c181bb80ea5fc9c1d7ae41bfe7e4a338ed36a1ccc9e8efd36509e" Oct 30 22:36:25.056062 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:36:25.056023 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4ce8ef7-kx9vm"] Oct 30 22:36:25.101660 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:36:25.101621 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4ce8ef7-kx9vm"] Oct 30 22:36:25.603472 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:36:25.603431 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="260a3f69-701a-4ff2-a2cd-769957f80522" path="/var/lib/kubelet/pods/260a3f69-701a-4ff2-a2cd-769957f80522/volumes" Oct 30 22:36:26.056784 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:36:26.056750 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd3d021e-97bzr"] Oct 30 22:36:26.103632 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:36:26.103586 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd3d021e-97bzr"] Oct 30 22:36:27.604810 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:36:27.604764 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8c456dc-1d23-456b-b272-1431f457b971" path="/var/lib/kubelet/pods/c8c456dc-1d23-456b-b272-1431f457b971/volumes" Oct 30 22:36:27.911831 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:36:27.911737 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:36:53.835478 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:36:53.835397 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:36:57.911609 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:36:57.911571 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:36:59.597389 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:36:59.597352 6006 scope.go:122] "RemoveContainer" containerID="aeb178d0a439b181855d5ad61291bca601eed05962bef15a69d4cd351e4fca81" Oct 30 22:37:22.411507 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:22.411465 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48bd57a2-8hvpj"] Oct 30 22:37:22.411941 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:22.411913 6006 cpu_manager.go:414] "containerMap: removing container" podUID="4660bcf0-9e67-41c3-b480-c6b25dfbc9b9" containerName="revoke-keys" Oct 30 22:37:22.411941 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:22.411934 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4660bcf0-9e67-41c3-b480-c6b25dfbc9b9" containerName="revoke-keys" Oct 30 22:37:22.412038 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:22.411984 6006 cpu_manager.go:414] "containerMap: removing container" podUID="2011ac7e-5d4d-4368-8a5a-55cdd5de789e" containerName="revoke-keys" Oct 30 22:37:22.412038 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:22.411990 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2011ac7e-5d4d-4368-8a5a-55cdd5de789e" containerName="revoke-keys" Oct 30 22:37:22.412107 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:22.412063 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="2011ac7e-5d4d-4368-8a5a-55cdd5de789e" containerName="revoke-keys" Oct 30 22:37:22.412107 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:22.412071 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="4660bcf0-9e67-41c3-b480-c6b25dfbc9b9" containerName="revoke-keys" Oct 30 22:37:22.416918 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:22.416893 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48bd57a2-8hvpj" Oct 30 22:37:22.473148 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:22.473103 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71d8d509-855b-4e9d-b9a0-8b62fbb40944-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48bd57a2-8hvpj\" (UID: \"71d8d509-855b-4e9d-b9a0-8b62fbb40944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48bd57a2-8hvpj" Oct 30 22:37:22.473357 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:22.473244 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71d8d509-855b-4e9d-b9a0-8b62fbb40944-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48bd57a2-8hvpj\" (UID: \"71d8d509-855b-4e9d-b9a0-8b62fbb40944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48bd57a2-8hvpj" Oct 30 22:37:22.473432 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:22.473414 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbxt5\" (UniqueName: \"kubernetes.io/projected/71d8d509-855b-4e9d-b9a0-8b62fbb40944-kube-api-access-pbxt5\") pod \"maas-api-revoke-sub-48bd57a2-8hvpj\" (UID: \"71d8d509-855b-4e9d-b9a0-8b62fbb40944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48bd57a2-8hvpj" Oct 30 22:37:22.515058 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:22.515009 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:37:22.515279 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:22.515088 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:37:22.515279 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:22.515176 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:37:22.515279 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:22.515205 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:37:22.518069 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:22.518041 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48bd57a2-8hvpj"] Oct 30 22:37:22.574069 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:22.574024 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbxt5\" (UniqueName: \"kubernetes.io/projected/71d8d509-855b-4e9d-b9a0-8b62fbb40944-kube-api-access-pbxt5\") pod \"maas-api-revoke-sub-48bd57a2-8hvpj\" (UID: \"71d8d509-855b-4e9d-b9a0-8b62fbb40944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48bd57a2-8hvpj" Oct 30 22:37:22.574307 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:22.574085 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71d8d509-855b-4e9d-b9a0-8b62fbb40944-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48bd57a2-8hvpj\" (UID: \"71d8d509-855b-4e9d-b9a0-8b62fbb40944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48bd57a2-8hvpj" Oct 30 22:37:22.574307 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:22.574185 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71d8d509-855b-4e9d-b9a0-8b62fbb40944-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48bd57a2-8hvpj\" (UID: \"71d8d509-855b-4e9d-b9a0-8b62fbb40944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48bd57a2-8hvpj" Oct 30 22:37:22.575104 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:22.575070 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71d8d509-855b-4e9d-b9a0-8b62fbb40944-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48bd57a2-8hvpj\" (UID: \"71d8d509-855b-4e9d-b9a0-8b62fbb40944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48bd57a2-8hvpj" Oct 30 22:37:22.577408 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:22.577389 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71d8d509-855b-4e9d-b9a0-8b62fbb40944-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48bd57a2-8hvpj\" (UID: \"71d8d509-855b-4e9d-b9a0-8b62fbb40944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48bd57a2-8hvpj" Oct 30 22:37:22.633013 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:22.632975 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbxt5\" (UniqueName: \"kubernetes.io/projected/71d8d509-855b-4e9d-b9a0-8b62fbb40944-kube-api-access-pbxt5\") pod \"maas-api-revoke-sub-48bd57a2-8hvpj\" (UID: \"71d8d509-855b-4e9d-b9a0-8b62fbb40944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48bd57a2-8hvpj" Oct 30 22:37:22.730617 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:22.730535 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48bd57a2-8hvpj" Oct 30 22:37:23.102772 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:23.102740 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48bd57a2-8hvpj"] Oct 30 22:37:23.105527 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:37:23.105496 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d8d509_855b_4e9d_b9a0_8b62fbb40944.slice/crio-7325120f078be557421981d8dae6b410ef9129c8a5b7deb62fbb18bd78865ea8 WatchSource:0}: Error finding container 7325120f078be557421981d8dae6b410ef9129c8a5b7deb62fbb18bd78865ea8: Status 404 returned error can't find the container with id 7325120f078be557421981d8dae6b410ef9129c8a5b7deb62fbb18bd78865ea8 Oct 30 22:37:23.876220 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:23.876181 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48bd57a2-8hvpj" event={"ID":"71d8d509-855b-4e9d-b9a0-8b62fbb40944","Type":"ContainerStarted","Data":"76becb12bf7e19163939d22d7834a6f9ae6601c1160ae0de3e98d48bc4ad1c8c"} Oct 30 22:37:23.876733 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:23.876243 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48bd57a2-8hvpj" event={"ID":"71d8d509-855b-4e9d-b9a0-8b62fbb40944","Type":"ContainerStarted","Data":"7325120f078be557421981d8dae6b410ef9129c8a5b7deb62fbb18bd78865ea8"} Oct 30 22:37:23.985918 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:23.985872 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48bd57a2-8hvpj" podStartSLOduration=1.985858603 podStartE2EDuration="1.985858603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:23.985788547 +0000 UTC m=+2846.537964846" watchObservedRunningTime="2026-10-30 22:37:23.985858603 +0000 UTC m=+2846.538034902" Oct 30 22:37:24.883814 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:24.883774 6006 generic.go:356] "Generic (PLEG): container finished" podID="71d8d509-855b-4e9d-b9a0-8b62fbb40944" containerID="76becb12bf7e19163939d22d7834a6f9ae6601c1160ae0de3e98d48bc4ad1c8c" exitCode=0 Oct 30 22:37:24.884290 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:24.883859 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48bd57a2-8hvpj" event={"ID":"71d8d509-855b-4e9d-b9a0-8b62fbb40944","Type":"ContainerDied","Data":"76becb12bf7e19163939d22d7834a6f9ae6601c1160ae0de3e98d48bc4ad1c8c"} Oct 30 22:37:26.034864 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:26.034831 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48bd57a2-8hvpj" Oct 30 22:37:26.116587 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:26.116554 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d8d509-855b-4e9d-b9a0-8b62fbb40944-kube-api-access-pbxt5\" (UniqueName: \"kubernetes.io/projected/71d8d509-855b-4e9d-b9a0-8b62fbb40944-kube-api-access-pbxt5\") pod \"71d8d509-855b-4e9d-b9a0-8b62fbb40944\" (UID: \"71d8d509-855b-4e9d-b9a0-8b62fbb40944\") " Oct 30 22:37:26.116794 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:26.116605 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71d8d509-855b-4e9d-b9a0-8b62fbb40944-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71d8d509-855b-4e9d-b9a0-8b62fbb40944-maas-api-openshift-ca\") pod \"71d8d509-855b-4e9d-b9a0-8b62fbb40944\" (UID: \"71d8d509-855b-4e9d-b9a0-8b62fbb40944\") " Oct 30 22:37:26.116794 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:26.116633 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71d8d509-855b-4e9d-b9a0-8b62fbb40944-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71d8d509-855b-4e9d-b9a0-8b62fbb40944-maas-api-xks-ca\") pod \"71d8d509-855b-4e9d-b9a0-8b62fbb40944\" (UID: \"71d8d509-855b-4e9d-b9a0-8b62fbb40944\") " Oct 30 22:37:26.117098 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:26.117063 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71d8d509-855b-4e9d-b9a0-8b62fbb40944-maas-api-openshift-ca" pod "71d8d509-855b-4e9d-b9a0-8b62fbb40944" (UID: "71d8d509-855b-4e9d-b9a0-8b62fbb40944"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:37:26.119487 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:26.119454 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d8d509-855b-4e9d-b9a0-8b62fbb40944-kube-api-access-pbxt5" pod "71d8d509-855b-4e9d-b9a0-8b62fbb40944" (UID: "71d8d509-855b-4e9d-b9a0-8b62fbb40944"). InnerVolumeSpecName "kube-api-access-pbxt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:26.119579 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:26.119466 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71d8d509-855b-4e9d-b9a0-8b62fbb40944-maas-api-xks-ca" pod "71d8d509-855b-4e9d-b9a0-8b62fbb40944" (UID: "71d8d509-855b-4e9d-b9a0-8b62fbb40944"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:37:26.218184 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:26.218084 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbxt5\" (UniqueName: \"kubernetes.io/projected/71d8d509-855b-4e9d-b9a0-8b62fbb40944-kube-api-access-pbxt5\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:26.218184 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:26.218122 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71d8d509-855b-4e9d-b9a0-8b62fbb40944-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:26.218184 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:26.218138 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71d8d509-855b-4e9d-b9a0-8b62fbb40944-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:26.896244 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:26.896212 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48bd57a2-8hvpj" Oct 30 22:37:26.896424 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:26.896261 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48bd57a2-8hvpj" event={"ID":"71d8d509-855b-4e9d-b9a0-8b62fbb40944","Type":"ContainerDied","Data":"7325120f078be557421981d8dae6b410ef9129c8a5b7deb62fbb18bd78865ea8"} Oct 30 22:37:26.896424 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:26.896317 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7325120f078be557421981d8dae6b410ef9129c8a5b7deb62fbb18bd78865ea8" Oct 30 22:37:27.911435 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:37:27.911385 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:37:35.433951 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:35.433911 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-225aacce-vfsft"] Oct 30 22:37:35.434669 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:35.434646 6006 cpu_manager.go:414] "containerMap: removing container" podUID="71d8d509-855b-4e9d-b9a0-8b62fbb40944" containerName="revoke-keys" Oct 30 22:37:35.434757 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:35.434672 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d8d509-855b-4e9d-b9a0-8b62fbb40944" containerName="revoke-keys" Oct 30 22:37:35.434817 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:35.434801 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d8d509-855b-4e9d-b9a0-8b62fbb40944" containerName="revoke-keys" Oct 30 22:37:35.438144 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:35.438120 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225aacce-vfsft" Oct 30 22:37:35.488849 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:35.488818 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:37:35.524945 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:35.524903 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3a268ec-2276-420c-8533-5a9d302381b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-225aacce-vfsft\" (UID: \"e3a268ec-2276-420c-8533-5a9d302381b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225aacce-vfsft" Oct 30 22:37:35.525144 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:35.524962 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkzjz\" (UniqueName: \"kubernetes.io/projected/e3a268ec-2276-420c-8533-5a9d302381b1-kube-api-access-bkzjz\") pod \"maas-api-revoke-sub-225aacce-vfsft\" (UID: \"e3a268ec-2276-420c-8533-5a9d302381b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225aacce-vfsft" Oct 30 22:37:35.525223 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:35.525143 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3a268ec-2276-420c-8533-5a9d302381b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-225aacce-vfsft\" (UID: \"e3a268ec-2276-420c-8533-5a9d302381b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225aacce-vfsft" Oct 30 22:37:35.535089 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:35.535055 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:37:35.535274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:35.535191 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:37:35.535356 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:35.535289 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:37:35.537385 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:35.537362 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-225aacce-vfsft"] Oct 30 22:37:35.626180 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:35.626147 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3a268ec-2276-420c-8533-5a9d302381b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-225aacce-vfsft\" (UID: \"e3a268ec-2276-420c-8533-5a9d302381b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225aacce-vfsft" Oct 30 22:37:35.626417 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:35.626193 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkzjz\" (UniqueName: \"kubernetes.io/projected/e3a268ec-2276-420c-8533-5a9d302381b1-kube-api-access-bkzjz\") pod \"maas-api-revoke-sub-225aacce-vfsft\" (UID: \"e3a268ec-2276-420c-8533-5a9d302381b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225aacce-vfsft" Oct 30 22:37:35.626417 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:35.626333 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3a268ec-2276-420c-8533-5a9d302381b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-225aacce-vfsft\" (UID: \"e3a268ec-2276-420c-8533-5a9d302381b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225aacce-vfsft" Oct 30 22:37:35.627056 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:35.627030 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3a268ec-2276-420c-8533-5a9d302381b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-225aacce-vfsft\" (UID: \"e3a268ec-2276-420c-8533-5a9d302381b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225aacce-vfsft" Oct 30 22:37:35.628609 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:35.628587 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3a268ec-2276-420c-8533-5a9d302381b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-225aacce-vfsft\" (UID: \"e3a268ec-2276-420c-8533-5a9d302381b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225aacce-vfsft" Oct 30 22:37:35.684617 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:35.684538 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkzjz\" (UniqueName: \"kubernetes.io/projected/e3a268ec-2276-420c-8533-5a9d302381b1-kube-api-access-bkzjz\") pod \"maas-api-revoke-sub-225aacce-vfsft\" (UID: \"e3a268ec-2276-420c-8533-5a9d302381b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-225aacce-vfsft" Oct 30 22:37:35.752378 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:35.752345 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225aacce-vfsft" Oct 30 22:37:36.123356 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:36.123324 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-225aacce-vfsft"] Oct 30 22:37:36.126042 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:37:36.126002 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a268ec_2276_420c_8533_5a9d302381b1.slice/crio-32c99a9df942c49236f763743d5417dee962a7d9b404e3977939951429c677ce WatchSource:0}: Error finding container 32c99a9df942c49236f763743d5417dee962a7d9b404e3977939951429c677ce: Status 404 returned error can't find the container with id 32c99a9df942c49236f763743d5417dee962a7d9b404e3977939951429c677ce Oct 30 22:37:36.958835 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:36.958793 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225aacce-vfsft" event={"ID":"e3a268ec-2276-420c-8533-5a9d302381b1","Type":"ContainerStarted","Data":"d1dee38f3771b6656d4ada3f72bed3ddb540ebd0566a31da8b238801ee8c41ff"} Oct 30 22:37:36.959302 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:36.958854 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225aacce-vfsft" event={"ID":"e3a268ec-2276-420c-8533-5a9d302381b1","Type":"ContainerStarted","Data":"32c99a9df942c49236f763743d5417dee962a7d9b404e3977939951429c677ce"} Oct 30 22:37:37.070088 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:37.070035 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225aacce-vfsft" podStartSLOduration=2.070018483 podStartE2EDuration="2.070018483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:37.069467677 +0000 UTC m=+2859.621643977" watchObservedRunningTime="2026-10-30 22:37:37.070018483 +0000 UTC m=+2859.622194787" Oct 30 22:37:37.966323 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:37.966287 6006 generic.go:356] "Generic (PLEG): container finished" podID="e3a268ec-2276-420c-8533-5a9d302381b1" containerID="d1dee38f3771b6656d4ada3f72bed3ddb540ebd0566a31da8b238801ee8c41ff" exitCode=0 Oct 30 22:37:37.966751 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:37.966421 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225aacce-vfsft" event={"ID":"e3a268ec-2276-420c-8533-5a9d302381b1","Type":"ContainerDied","Data":"d1dee38f3771b6656d4ada3f72bed3ddb540ebd0566a31da8b238801ee8c41ff"} Oct 30 22:37:39.128496 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:39.128463 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225aacce-vfsft" Oct 30 22:37:39.264526 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:39.264444 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3a268ec-2276-420c-8533-5a9d302381b1-kube-api-access-bkzjz\" (UniqueName: \"kubernetes.io/projected/e3a268ec-2276-420c-8533-5a9d302381b1-kube-api-access-bkzjz\") pod \"e3a268ec-2276-420c-8533-5a9d302381b1\" (UID: \"e3a268ec-2276-420c-8533-5a9d302381b1\") " Oct 30 22:37:39.264697 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:39.264528 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3a268ec-2276-420c-8533-5a9d302381b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3a268ec-2276-420c-8533-5a9d302381b1-maas-api-openshift-ca\") pod \"e3a268ec-2276-420c-8533-5a9d302381b1\" (UID: \"e3a268ec-2276-420c-8533-5a9d302381b1\") " Oct 30 22:37:39.264697 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:39.264634 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3a268ec-2276-420c-8533-5a9d302381b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3a268ec-2276-420c-8533-5a9d302381b1-maas-api-xks-ca\") pod \"e3a268ec-2276-420c-8533-5a9d302381b1\" (UID: \"e3a268ec-2276-420c-8533-5a9d302381b1\") " Oct 30 22:37:39.264996 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:39.264959 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3a268ec-2276-420c-8533-5a9d302381b1-maas-api-openshift-ca" pod "e3a268ec-2276-420c-8533-5a9d302381b1" (UID: "e3a268ec-2276-420c-8533-5a9d302381b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:37:39.266647 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:39.266617 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3a268ec-2276-420c-8533-5a9d302381b1-maas-api-xks-ca" pod "e3a268ec-2276-420c-8533-5a9d302381b1" (UID: "e3a268ec-2276-420c-8533-5a9d302381b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:37:39.266851 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:39.266825 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3a268ec-2276-420c-8533-5a9d302381b1-kube-api-access-bkzjz" pod "e3a268ec-2276-420c-8533-5a9d302381b1" (UID: "e3a268ec-2276-420c-8533-5a9d302381b1"). InnerVolumeSpecName "kube-api-access-bkzjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:39.365992 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:39.365950 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkzjz\" (UniqueName: \"kubernetes.io/projected/e3a268ec-2276-420c-8533-5a9d302381b1-kube-api-access-bkzjz\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:39.365992 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:39.365991 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3a268ec-2276-420c-8533-5a9d302381b1-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:39.365992 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:39.366003 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3a268ec-2276-420c-8533-5a9d302381b1-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:39.979216 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:39.979138 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225aacce-vfsft" Oct 30 22:37:39.979216 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:39.979174 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-225aacce-vfsft" event={"ID":"e3a268ec-2276-420c-8533-5a9d302381b1","Type":"ContainerDied","Data":"32c99a9df942c49236f763743d5417dee962a7d9b404e3977939951429c677ce"} Oct 30 22:37:39.979442 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:39.979226 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32c99a9df942c49236f763743d5417dee962a7d9b404e3977939951429c677ce" Oct 30 22:37:46.920192 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:46.920146 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:37:57.909949 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:37:57.909909 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:37:59.623307 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:59.623273 6006 scope.go:122] "RemoveContainer" containerID="618d9285f2336da155626be28220ec8dcc94a78206a2e86751a3c4ea7910daa2" Oct 30 22:37:59.633499 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:59.633471 6006 scope.go:122] "RemoveContainer" containerID="ca106e78a3ecf3e0f5b77950388ae40d61151e203a5eb2e68a55e7673b49bebf" Oct 30 22:37:59.642859 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:59.642828 6006 scope.go:122] "RemoveContainer" containerID="f42cc628c94ec5d9decfa8291ba1936e5145410ab078678466660ce9e218a7bc" Oct 30 22:37:59.652461 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:37:59.652440 6006 scope.go:122] "RemoveContainer" containerID="0543281f09eab4f1e5e9d7e83ff3f62aa08a6b2322355f8d3c2593aeac51897d" Oct 30 22:38:03.775141 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:03.775106 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-582593-j4dmd"] Oct 30 22:38:03.775600 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:03.775585 6006 cpu_manager.go:414] "containerMap: removing container" podUID="e3a268ec-2276-420c-8533-5a9d302381b1" containerName="revoke-keys" Oct 30 22:38:03.775657 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:03.775601 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3a268ec-2276-420c-8533-5a9d302381b1" containerName="revoke-keys" Oct 30 22:38:03.775701 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:03.775692 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3a268ec-2276-420c-8533-5a9d302381b1" containerName="revoke-keys" Oct 30 22:38:03.787518 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:03.787483 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-582593-j4dmd" Oct 30 22:38:03.822579 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:03.822540 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21cdb1b9-31dc-42a9-bcc4-706f7628237e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-582593-j4dmd\" (UID: \"21cdb1b9-31dc-42a9-bcc4-706f7628237e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-582593-j4dmd" Oct 30 22:38:03.822775 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:03.822601 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvwjx\" (UniqueName: \"kubernetes.io/projected/21cdb1b9-31dc-42a9-bcc4-706f7628237e-kube-api-access-mvwjx\") pod \"maas-api-revoke-keys-e2e-inf-a-582593-j4dmd\" (UID: \"21cdb1b9-31dc-42a9-bcc4-706f7628237e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-582593-j4dmd" Oct 30 22:38:03.822841 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:03.822816 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21cdb1b9-31dc-42a9-bcc4-706f7628237e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-582593-j4dmd\" (UID: \"21cdb1b9-31dc-42a9-bcc4-706f7628237e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-582593-j4dmd" Oct 30 22:38:03.834775 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:03.834735 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-582593-j4dmd"] Oct 30 22:38:03.878336 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:03.878297 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:38:03.878336 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:03.878327 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:38:03.882798 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:03.882767 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:38:03.882924 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:03.882801 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:38:03.924379 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:03.924344 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21cdb1b9-31dc-42a9-bcc4-706f7628237e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-582593-j4dmd\" (UID: \"21cdb1b9-31dc-42a9-bcc4-706f7628237e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-582593-j4dmd" Oct 30 22:38:03.924570 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:03.924387 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvwjx\" (UniqueName: \"kubernetes.io/projected/21cdb1b9-31dc-42a9-bcc4-706f7628237e-kube-api-access-mvwjx\") pod \"maas-api-revoke-keys-e2e-inf-a-582593-j4dmd\" (UID: \"21cdb1b9-31dc-42a9-bcc4-706f7628237e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-582593-j4dmd" Oct 30 22:38:03.924570 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:03.924528 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21cdb1b9-31dc-42a9-bcc4-706f7628237e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-582593-j4dmd\" (UID: \"21cdb1b9-31dc-42a9-bcc4-706f7628237e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-582593-j4dmd" Oct 30 22:38:03.925092 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:03.925070 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21cdb1b9-31dc-42a9-bcc4-706f7628237e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-582593-j4dmd\" (UID: \"21cdb1b9-31dc-42a9-bcc4-706f7628237e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-582593-j4dmd" Oct 30 22:38:03.926962 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:03.926929 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21cdb1b9-31dc-42a9-bcc4-706f7628237e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-582593-j4dmd\" (UID: \"21cdb1b9-31dc-42a9-bcc4-706f7628237e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-582593-j4dmd" Oct 30 22:38:03.984693 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:03.984655 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvwjx\" (UniqueName: \"kubernetes.io/projected/21cdb1b9-31dc-42a9-bcc4-706f7628237e-kube-api-access-mvwjx\") pod \"maas-api-revoke-keys-e2e-inf-a-582593-j4dmd\" (UID: \"21cdb1b9-31dc-42a9-bcc4-706f7628237e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-582593-j4dmd" Oct 30 22:38:04.103677 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:04.103639 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-582593-j4dmd" Oct 30 22:38:04.475133 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:04.475099 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-582593-j4dmd"] Oct 30 22:38:04.477604 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:38:04.477573 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21cdb1b9_31dc_42a9_bcc4_706f7628237e.slice/crio-0fdfe937f3ec51d8c41459f40bb5348ca834443b6cf0f0a6a11cebc529747e24 WatchSource:0}: Error finding container 0fdfe937f3ec51d8c41459f40bb5348ca834443b6cf0f0a6a11cebc529747e24: Status 404 returned error can't find the container with id 0fdfe937f3ec51d8c41459f40bb5348ca834443b6cf0f0a6a11cebc529747e24 Oct 30 22:38:05.132969 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:05.132876 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-582593-j4dmd" event={"ID":"21cdb1b9-31dc-42a9-bcc4-706f7628237e","Type":"ContainerStarted","Data":"e878eb9184c8a7bbc1c91e09a44e81ad6c2b1d5e9124f2eb3b0487939ad2f6a0"} Oct 30 22:38:05.132969 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:05.132940 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-582593-j4dmd" event={"ID":"21cdb1b9-31dc-42a9-bcc4-706f7628237e","Type":"ContainerStarted","Data":"0fdfe937f3ec51d8c41459f40bb5348ca834443b6cf0f0a6a11cebc529747e24"} Oct 30 22:38:05.242863 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:05.242804 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-582593-j4dmd" podStartSLOduration=2.242789182 podStartE2EDuration="2.242789182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:05.242483622 +0000 UTC m=+2887.794659922" watchObservedRunningTime="2026-10-30 22:38:05.242789182 +0000 UTC m=+2887.794965479" Oct 30 22:38:06.145453 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:06.145412 6006 generic.go:356] "Generic (PLEG): container finished" podID="21cdb1b9-31dc-42a9-bcc4-706f7628237e" containerID="e878eb9184c8a7bbc1c91e09a44e81ad6c2b1d5e9124f2eb3b0487939ad2f6a0" exitCode=0 Oct 30 22:38:06.146180 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:06.146151 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-582593-j4dmd" event={"ID":"21cdb1b9-31dc-42a9-bcc4-706f7628237e","Type":"ContainerDied","Data":"e878eb9184c8a7bbc1c91e09a44e81ad6c2b1d5e9124f2eb3b0487939ad2f6a0"} Oct 30 22:38:07.305129 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:07.305101 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-582593-j4dmd" Oct 30 22:38:07.358976 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:07.358936 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21cdb1b9-31dc-42a9-bcc4-706f7628237e-kube-api-access-mvwjx\" (UniqueName: \"kubernetes.io/projected/21cdb1b9-31dc-42a9-bcc4-706f7628237e-kube-api-access-mvwjx\") pod \"21cdb1b9-31dc-42a9-bcc4-706f7628237e\" (UID: \"21cdb1b9-31dc-42a9-bcc4-706f7628237e\") " Oct 30 22:38:07.359172 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:07.359010 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21cdb1b9-31dc-42a9-bcc4-706f7628237e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21cdb1b9-31dc-42a9-bcc4-706f7628237e-maas-api-xks-ca\") pod \"21cdb1b9-31dc-42a9-bcc4-706f7628237e\" (UID: \"21cdb1b9-31dc-42a9-bcc4-706f7628237e\") " Oct 30 22:38:07.359172 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:07.359094 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21cdb1b9-31dc-42a9-bcc4-706f7628237e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21cdb1b9-31dc-42a9-bcc4-706f7628237e-maas-api-openshift-ca\") pod \"21cdb1b9-31dc-42a9-bcc4-706f7628237e\" (UID: \"21cdb1b9-31dc-42a9-bcc4-706f7628237e\") " Oct 30 22:38:07.359482 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:07.359457 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21cdb1b9-31dc-42a9-bcc4-706f7628237e-maas-api-openshift-ca" pod "21cdb1b9-31dc-42a9-bcc4-706f7628237e" (UID: "21cdb1b9-31dc-42a9-bcc4-706f7628237e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:38:07.361183 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:07.361162 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21cdb1b9-31dc-42a9-bcc4-706f7628237e-kube-api-access-mvwjx" pod "21cdb1b9-31dc-42a9-bcc4-706f7628237e" (UID: "21cdb1b9-31dc-42a9-bcc4-706f7628237e"). InnerVolumeSpecName "kube-api-access-mvwjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:07.361285 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:07.361161 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21cdb1b9-31dc-42a9-bcc4-706f7628237e-maas-api-xks-ca" pod "21cdb1b9-31dc-42a9-bcc4-706f7628237e" (UID: "21cdb1b9-31dc-42a9-bcc4-706f7628237e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:07.460506 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:07.460422 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvwjx\" (UniqueName: \"kubernetes.io/projected/21cdb1b9-31dc-42a9-bcc4-706f7628237e-kube-api-access-mvwjx\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:07.460506 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:07.460451 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21cdb1b9-31dc-42a9-bcc4-706f7628237e-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:07.460506 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:07.460461 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21cdb1b9-31dc-42a9-bcc4-706f7628237e-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:08.158674 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:08.158635 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-582593-j4dmd" event={"ID":"21cdb1b9-31dc-42a9-bcc4-706f7628237e","Type":"ContainerDied","Data":"0fdfe937f3ec51d8c41459f40bb5348ca834443b6cf0f0a6a11cebc529747e24"} Oct 30 22:38:08.158890 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:08.158692 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fdfe937f3ec51d8c41459f40bb5348ca834443b6cf0f0a6a11cebc529747e24" Oct 30 22:38:08.158890 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:08.158645 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-582593-j4dmd" Oct 30 22:38:11.101460 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:11.101423 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:38:14.241643 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:14.241600 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92"] Oct 30 22:38:16.175026 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:16.174988 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f053c1c1-hxg8j"] Oct 30 22:38:16.175475 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:16.175442 6006 cpu_manager.go:414] "containerMap: removing container" podUID="21cdb1b9-31dc-42a9-bcc4-706f7628237e" containerName="revoke-keys" Oct 30 22:38:16.175475 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:16.175455 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21cdb1b9-31dc-42a9-bcc4-706f7628237e" containerName="revoke-keys" Oct 30 22:38:16.175612 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:16.175565 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="21cdb1b9-31dc-42a9-bcc4-706f7628237e" containerName="revoke-keys" Oct 30 22:38:16.178641 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:16.178620 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f053c1c1-hxg8j" Oct 30 22:38:16.228337 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:16.228294 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:38:16.228337 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:16.228321 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:38:16.228337 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:16.228322 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:38:16.228587 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:16.228443 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:38:16.267522 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:16.267485 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27986857-6552-4518-bc63-da753984fc2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f053c1c1-hxg8j\" (UID: \"27986857-6552-4518-bc63-da753984fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f053c1c1-hxg8j" Oct 30 22:38:16.267710 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:16.267629 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2k82\" (UniqueName: \"kubernetes.io/projected/27986857-6552-4518-bc63-da753984fc2e-kube-api-access-l2k82\") pod \"maas-api-revoke-sub-f053c1c1-hxg8j\" (UID: \"27986857-6552-4518-bc63-da753984fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f053c1c1-hxg8j" Oct 30 22:38:16.267795 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:16.267769 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27986857-6552-4518-bc63-da753984fc2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f053c1c1-hxg8j\" (UID: \"27986857-6552-4518-bc63-da753984fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f053c1c1-hxg8j" Oct 30 22:38:16.278661 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:16.278628 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f053c1c1-hxg8j"] Oct 30 22:38:16.368800 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:16.368765 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27986857-6552-4518-bc63-da753984fc2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f053c1c1-hxg8j\" (UID: \"27986857-6552-4518-bc63-da753984fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f053c1c1-hxg8j" Oct 30 22:38:16.368986 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:16.368859 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27986857-6552-4518-bc63-da753984fc2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f053c1c1-hxg8j\" (UID: \"27986857-6552-4518-bc63-da753984fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f053c1c1-hxg8j" Oct 30 22:38:16.368986 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:16.368928 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2k82\" (UniqueName: \"kubernetes.io/projected/27986857-6552-4518-bc63-da753984fc2e-kube-api-access-l2k82\") pod \"maas-api-revoke-sub-f053c1c1-hxg8j\" (UID: \"27986857-6552-4518-bc63-da753984fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f053c1c1-hxg8j" Oct 30 22:38:16.369589 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:16.369560 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27986857-6552-4518-bc63-da753984fc2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f053c1c1-hxg8j\" (UID: \"27986857-6552-4518-bc63-da753984fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f053c1c1-hxg8j" Oct 30 22:38:16.371383 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:16.371360 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27986857-6552-4518-bc63-da753984fc2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f053c1c1-hxg8j\" (UID: \"27986857-6552-4518-bc63-da753984fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f053c1c1-hxg8j" Oct 30 22:38:16.427813 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:16.427736 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2k82\" (UniqueName: \"kubernetes.io/projected/27986857-6552-4518-bc63-da753984fc2e-kube-api-access-l2k82\") pod \"maas-api-revoke-sub-f053c1c1-hxg8j\" (UID: \"27986857-6552-4518-bc63-da753984fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f053c1c1-hxg8j" Oct 30 22:38:16.491910 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:16.491876 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f053c1c1-hxg8j" Oct 30 22:38:16.870523 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:16.870481 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f053c1c1-hxg8j"] Oct 30 22:38:16.874377 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:38:16.874348 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27986857_6552_4518_bc63_da753984fc2e.slice/crio-6d63e1fef3a9c2a3a9e2c165cfc289672925aa3570abe8b6bd2cd42b34d42f98 WatchSource:0}: Error finding container 6d63e1fef3a9c2a3a9e2c165cfc289672925aa3570abe8b6bd2cd42b34d42f98: Status 404 returned error can't find the container with id 6d63e1fef3a9c2a3a9e2c165cfc289672925aa3570abe8b6bd2cd42b34d42f98 Oct 30 22:38:17.219896 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:17.219864 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f053c1c1-hxg8j" event={"ID":"27986857-6552-4518-bc63-da753984fc2e","Type":"ContainerStarted","Data":"b5d6d6b1e80f937c6429b7f3037f309f2497b9c41b2224ad49432e578e7b669c"} Oct 30 22:38:17.220281 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:17.219916 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f053c1c1-hxg8j" event={"ID":"27986857-6552-4518-bc63-da753984fc2e","Type":"ContainerStarted","Data":"6d63e1fef3a9c2a3a9e2c165cfc289672925aa3570abe8b6bd2cd42b34d42f98"} Oct 30 22:38:18.227314 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:18.227240 6006 generic.go:356] "Generic (PLEG): container finished" podID="27986857-6552-4518-bc63-da753984fc2e" containerID="b5d6d6b1e80f937c6429b7f3037f309f2497b9c41b2224ad49432e578e7b669c" exitCode=0 Oct 30 22:38:18.227727 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:18.227324 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f053c1c1-hxg8j" event={"ID":"27986857-6552-4518-bc63-da753984fc2e","Type":"ContainerDied","Data":"b5d6d6b1e80f937c6429b7f3037f309f2497b9c41b2224ad49432e578e7b669c"} Oct 30 22:38:19.243359 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:19.243302 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92" podUID="c8a428e4-1d5b-452d-a330-5d9c22c7d1f5" containerName="payload-pre-processing" containerID="cri-o://0fe175fb65c0dd00ee202ef571ed3bba3b246bfa858cf2edee03e6ed05959b73" gracePeriod=25 Oct 30 22:38:19.389989 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:19.389958 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f053c1c1-hxg8j" Oct 30 22:38:19.499918 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:19.499810 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27986857-6552-4518-bc63-da753984fc2e-kube-api-access-l2k82\" (UniqueName: \"kubernetes.io/projected/27986857-6552-4518-bc63-da753984fc2e-kube-api-access-l2k82\") pod \"27986857-6552-4518-bc63-da753984fc2e\" (UID: \"27986857-6552-4518-bc63-da753984fc2e\") " Oct 30 22:38:19.500099 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:19.499949 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27986857-6552-4518-bc63-da753984fc2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27986857-6552-4518-bc63-da753984fc2e-maas-api-xks-ca\") pod \"27986857-6552-4518-bc63-da753984fc2e\" (UID: \"27986857-6552-4518-bc63-da753984fc2e\") " Oct 30 22:38:19.500537 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:19.500291 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27986857-6552-4518-bc63-da753984fc2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27986857-6552-4518-bc63-da753984fc2e-maas-api-openshift-ca\") pod \"27986857-6552-4518-bc63-da753984fc2e\" (UID: \"27986857-6552-4518-bc63-da753984fc2e\") " Oct 30 22:38:19.500537 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:19.500517 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27986857-6552-4518-bc63-da753984fc2e-maas-api-openshift-ca" pod "27986857-6552-4518-bc63-da753984fc2e" (UID: "27986857-6552-4518-bc63-da753984fc2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:38:19.501257 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:19.501225 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27986857-6552-4518-bc63-da753984fc2e-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:19.506786 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:19.503477 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27986857-6552-4518-bc63-da753984fc2e-kube-api-access-l2k82" pod "27986857-6552-4518-bc63-da753984fc2e" (UID: "27986857-6552-4518-bc63-da753984fc2e"). InnerVolumeSpecName "kube-api-access-l2k82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:19.506786 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:19.505668 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27986857-6552-4518-bc63-da753984fc2e-maas-api-xks-ca" pod "27986857-6552-4518-bc63-da753984fc2e" (UID: "27986857-6552-4518-bc63-da753984fc2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:19.601986 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:19.601951 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27986857-6552-4518-bc63-da753984fc2e-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:19.601986 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:19.601981 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2k82\" (UniqueName: \"kubernetes.io/projected/27986857-6552-4518-bc63-da753984fc2e-kube-api-access-l2k82\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:20.241116 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:20.241080 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f053c1c1-hxg8j" Oct 30 22:38:20.241320 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:20.241115 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f053c1c1-hxg8j" event={"ID":"27986857-6552-4518-bc63-da753984fc2e","Type":"ContainerDied","Data":"6d63e1fef3a9c2a3a9e2c165cfc289672925aa3570abe8b6bd2cd42b34d42f98"} Oct 30 22:38:20.241320 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:20.241176 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d63e1fef3a9c2a3a9e2c165cfc289672925aa3570abe8b6bd2cd42b34d42f98" Oct 30 22:38:21.047861 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:21.047822 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc37f9f4-7mzcb"] Oct 30 22:38:21.094651 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:21.094615 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86e37fb2-tpm9t"] Oct 30 22:38:21.096458 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:21.096429 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc37f9f4-7mzcb"] Oct 30 22:38:21.097992 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:21.097968 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86e37fb2-tpm9t"] Oct 30 22:38:21.603032 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:21.602995 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16f728f0-4dce-43c2-8e6d-c2c38ee44c63" path="/var/lib/kubelet/pods/16f728f0-4dce-43c2-8e6d-c2c38ee44c63/volumes" Oct 30 22:38:21.603550 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:21.603526 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4adb785-11b7-44a1-a95d-c5ecac657922" path="/var/lib/kubelet/pods/b4adb785-11b7-44a1-a95d-c5ecac657922/volumes" Oct 30 22:38:23.987505 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:23.987459 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92" podUID="c8a428e4-1d5b-452d-a330-5d9c22c7d1f5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:38:27.911314 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:38:27.911268 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:38:33.987451 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:33.987406 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92" podUID="c8a428e4-1d5b-452d-a330-5d9c22c7d1f5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:38:35.737840 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:35.737803 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-582593-j4dmd"] Oct 30 22:38:35.784344 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:35.784306 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-582593-j4dmd"] Oct 30 22:38:37.604580 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:37.604536 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21cdb1b9-31dc-42a9-bcc4-706f7628237e" path="/var/lib/kubelet/pods/21cdb1b9-31dc-42a9-bcc4-706f7628237e/volumes" Oct 30 22:38:39.391159 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:39.391118 6006 generic.go:356] "Generic (PLEG): container finished" podID="c8a428e4-1d5b-452d-a330-5d9c22c7d1f5" containerID="0fe175fb65c0dd00ee202ef571ed3bba3b246bfa858cf2edee03e6ed05959b73" exitCode=0 Oct 30 22:38:39.391745 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:39.391194 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92" event={"ID":"c8a428e4-1d5b-452d-a330-5d9c22c7d1f5","Type":"ContainerDied","Data":"0fe175fb65c0dd00ee202ef571ed3bba3b246bfa858cf2edee03e6ed05959b73"} Oct 30 22:38:39.410361 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:39.408823 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz"] Oct 30 22:38:39.592052 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:39.592021 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92" Oct 30 22:38:39.616496 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:39.616457 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:38:39.734060 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:39.734020 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8a428e4-1d5b-452d-a330-5d9c22c7d1f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8a428e4-1d5b-452d-a330-5d9c22c7d1f5-plugins-config-volume\") pod \"c8a428e4-1d5b-452d-a330-5d9c22c7d1f5\" (UID: \"c8a428e4-1d5b-452d-a330-5d9c22c7d1f5\") " Oct 30 22:38:39.734263 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:39.734172 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a428e4-1d5b-452d-a330-5d9c22c7d1f5-kube-api-access-zkvxx\" (UniqueName: \"kubernetes.io/projected/c8a428e4-1d5b-452d-a330-5d9c22c7d1f5-kube-api-access-zkvxx\") pod \"c8a428e4-1d5b-452d-a330-5d9c22c7d1f5\" (UID: \"c8a428e4-1d5b-452d-a330-5d9c22c7d1f5\") " Oct 30 22:38:39.734555 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:39.734524 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a428e4-1d5b-452d-a330-5d9c22c7d1f5-plugins-config-volume" pod "c8a428e4-1d5b-452d-a330-5d9c22c7d1f5" (UID: "c8a428e4-1d5b-452d-a330-5d9c22c7d1f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:38:39.734739 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:39.734714 6006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8a428e4-1d5b-452d-a330-5d9c22c7d1f5-plugins-config-volume\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:39.737817 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:39.737784 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a428e4-1d5b-452d-a330-5d9c22c7d1f5-kube-api-access-zkvxx" pod "c8a428e4-1d5b-452d-a330-5d9c22c7d1f5" (UID: "c8a428e4-1d5b-452d-a330-5d9c22c7d1f5"). InnerVolumeSpecName "kube-api-access-zkvxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:39.835869 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:39.835768 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkvxx\" (UniqueName: \"kubernetes.io/projected/c8a428e4-1d5b-452d-a330-5d9c22c7d1f5-kube-api-access-zkvxx\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:40.398672 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.398628 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92" event={"ID":"c8a428e4-1d5b-452d-a330-5d9c22c7d1f5","Type":"ContainerDied","Data":"bc52ac1c84ecd404a2fcc70f01e592089df597f499151fb214ff3d39c5714950"} Oct 30 22:38:40.399128 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.398686 6006 scope.go:122] "RemoveContainer" containerID="0fe175fb65c0dd00ee202ef571ed3bba3b246bfa858cf2edee03e6ed05959b73" Oct 30 22:38:40.399128 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.398648 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92" Oct 30 22:38:40.414775 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.414747 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32401d4a-6xph4"] Oct 30 22:38:40.415201 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.415189 6006 cpu_manager.go:414] "containerMap: removing container" podUID="27986857-6552-4518-bc63-da753984fc2e" containerName="revoke-keys" Oct 30 22:38:40.415899 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.415202 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27986857-6552-4518-bc63-da753984fc2e" containerName="revoke-keys" Oct 30 22:38:40.415899 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.415216 6006 cpu_manager.go:414] "containerMap: removing container" podUID="c8a428e4-1d5b-452d-a330-5d9c22c7d1f5" containerName="payload-pre-processing" Oct 30 22:38:40.415899 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.415221 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a428e4-1d5b-452d-a330-5d9c22c7d1f5" containerName="payload-pre-processing" Oct 30 22:38:40.415899 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.415319 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="27986857-6552-4518-bc63-da753984fc2e" containerName="revoke-keys" Oct 30 22:38:40.415899 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.415329 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a428e4-1d5b-452d-a330-5d9c22c7d1f5" containerName="payload-pre-processing" Oct 30 22:38:40.419740 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.419722 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32401d4a-6xph4" Oct 30 22:38:40.471670 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.471628 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:38:40.475039 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.475011 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:38:40.501138 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.501105 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:38:40.501138 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.501154 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:38:40.504765 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.504733 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32401d4a-6xph4"] Oct 30 22:38:40.543132 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.543086 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78lbr\" (UniqueName: \"kubernetes.io/projected/03b0d538-05a6-4897-aae0-85f5ccc21dd0-kube-api-access-78lbr\") pod \"maas-api-revoke-sub-32401d4a-6xph4\" (UID: \"03b0d538-05a6-4897-aae0-85f5ccc21dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32401d4a-6xph4" Oct 30 22:38:40.543362 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.543299 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b0d538-05a6-4897-aae0-85f5ccc21dd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32401d4a-6xph4\" (UID: \"03b0d538-05a6-4897-aae0-85f5ccc21dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32401d4a-6xph4" Oct 30 22:38:40.543362 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.543356 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b0d538-05a6-4897-aae0-85f5ccc21dd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32401d4a-6xph4\" (UID: \"03b0d538-05a6-4897-aae0-85f5ccc21dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32401d4a-6xph4" Oct 30 22:38:40.588558 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.588520 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92"] Oct 30 22:38:40.595120 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.595091 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-582593-687978c77f-d7p92"] Oct 30 22:38:40.644773 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.644737 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b0d538-05a6-4897-aae0-85f5ccc21dd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32401d4a-6xph4\" (UID: \"03b0d538-05a6-4897-aae0-85f5ccc21dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32401d4a-6xph4" Oct 30 22:38:40.644773 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.644779 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b0d538-05a6-4897-aae0-85f5ccc21dd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32401d4a-6xph4\" (UID: \"03b0d538-05a6-4897-aae0-85f5ccc21dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32401d4a-6xph4" Oct 30 22:38:40.645047 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.644844 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78lbr\" (UniqueName: \"kubernetes.io/projected/03b0d538-05a6-4897-aae0-85f5ccc21dd0-kube-api-access-78lbr\") pod \"maas-api-revoke-sub-32401d4a-6xph4\" (UID: \"03b0d538-05a6-4897-aae0-85f5ccc21dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32401d4a-6xph4" Oct 30 22:38:40.645617 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.645589 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b0d538-05a6-4897-aae0-85f5ccc21dd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32401d4a-6xph4\" (UID: \"03b0d538-05a6-4897-aae0-85f5ccc21dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32401d4a-6xph4" Oct 30 22:38:40.647344 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.647317 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b0d538-05a6-4897-aae0-85f5ccc21dd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32401d4a-6xph4\" (UID: \"03b0d538-05a6-4897-aae0-85f5ccc21dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32401d4a-6xph4" Oct 30 22:38:40.706735 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.706653 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78lbr\" (UniqueName: \"kubernetes.io/projected/03b0d538-05a6-4897-aae0-85f5ccc21dd0-kube-api-access-78lbr\") pod \"maas-api-revoke-sub-32401d4a-6xph4\" (UID: \"03b0d538-05a6-4897-aae0-85f5ccc21dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32401d4a-6xph4" Oct 30 22:38:40.732921 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:40.732884 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32401d4a-6xph4" Oct 30 22:38:41.105685 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:41.105660 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32401d4a-6xph4"] Oct 30 22:38:41.107731 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:38:41.107699 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b0d538_05a6_4897_aae0_85f5ccc21dd0.slice/crio-fc3354933d6543740bee84cb1d02ebf962ad21c0dc832051713b7f89ef1308e9 WatchSource:0}: Error finding container fc3354933d6543740bee84cb1d02ebf962ad21c0dc832051713b7f89ef1308e9: Status 404 returned error can't find the container with id fc3354933d6543740bee84cb1d02ebf962ad21c0dc832051713b7f89ef1308e9 Oct 30 22:38:41.406310 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:41.406277 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32401d4a-6xph4" event={"ID":"03b0d538-05a6-4897-aae0-85f5ccc21dd0","Type":"ContainerStarted","Data":"fc3354933d6543740bee84cb1d02ebf962ad21c0dc832051713b7f89ef1308e9"} Oct 30 22:38:41.603575 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:41.603534 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8a428e4-1d5b-452d-a330-5d9c22c7d1f5" path="/var/lib/kubelet/pods/c8a428e4-1d5b-452d-a330-5d9c22c7d1f5/volumes" Oct 30 22:38:42.415031 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:42.414933 6006 generic.go:356] "Generic (PLEG): container finished" podID="03b0d538-05a6-4897-aae0-85f5ccc21dd0" containerID="3b40aa92ce424e673d8adc31f118f823699c6130696560531e3c8f40671e2d6a" exitCode=0 Oct 30 22:38:42.415518 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:42.415024 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32401d4a-6xph4" event={"ID":"03b0d538-05a6-4897-aae0-85f5ccc21dd0","Type":"ContainerDied","Data":"3b40aa92ce424e673d8adc31f118f823699c6130696560531e3c8f40671e2d6a"} Oct 30 22:38:43.582288 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:43.582235 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32401d4a-6xph4" Oct 30 22:38:43.676150 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:43.676041 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03b0d538-05a6-4897-aae0-85f5ccc21dd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b0d538-05a6-4897-aae0-85f5ccc21dd0-maas-api-openshift-ca\") pod \"03b0d538-05a6-4897-aae0-85f5ccc21dd0\" (UID: \"03b0d538-05a6-4897-aae0-85f5ccc21dd0\") " Oct 30 22:38:43.676334 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:43.676195 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03b0d538-05a6-4897-aae0-85f5ccc21dd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b0d538-05a6-4897-aae0-85f5ccc21dd0-maas-api-xks-ca\") pod \"03b0d538-05a6-4897-aae0-85f5ccc21dd0\" (UID: \"03b0d538-05a6-4897-aae0-85f5ccc21dd0\") " Oct 30 22:38:43.676334 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:43.676268 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03b0d538-05a6-4897-aae0-85f5ccc21dd0-kube-api-access-78lbr\" (UniqueName: \"kubernetes.io/projected/03b0d538-05a6-4897-aae0-85f5ccc21dd0-kube-api-access-78lbr\") pod \"03b0d538-05a6-4897-aae0-85f5ccc21dd0\" (UID: \"03b0d538-05a6-4897-aae0-85f5ccc21dd0\") " Oct 30 22:38:43.676507 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:43.676477 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b0d538-05a6-4897-aae0-85f5ccc21dd0-maas-api-openshift-ca" pod "03b0d538-05a6-4897-aae0-85f5ccc21dd0" (UID: "03b0d538-05a6-4897-aae0-85f5ccc21dd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:38:43.678322 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:43.678291 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03b0d538-05a6-4897-aae0-85f5ccc21dd0-maas-api-xks-ca" pod "03b0d538-05a6-4897-aae0-85f5ccc21dd0" (UID: "03b0d538-05a6-4897-aae0-85f5ccc21dd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:43.678451 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:43.678344 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b0d538-05a6-4897-aae0-85f5ccc21dd0-kube-api-access-78lbr" pod "03b0d538-05a6-4897-aae0-85f5ccc21dd0" (UID: "03b0d538-05a6-4897-aae0-85f5ccc21dd0"). InnerVolumeSpecName "kube-api-access-78lbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:43.777383 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:43.777331 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b0d538-05a6-4897-aae0-85f5ccc21dd0-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:43.777383 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:43.777373 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78lbr\" (UniqueName: \"kubernetes.io/projected/03b0d538-05a6-4897-aae0-85f5ccc21dd0-kube-api-access-78lbr\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:43.777383 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:43.777388 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b0d538-05a6-4897-aae0-85f5ccc21dd0-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:44.072588 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:44.072554 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-506964-pkw4b"] Oct 30 22:38:44.073044 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:44.073021 6006 cpu_manager.go:414] "containerMap: removing container" podUID="03b0d538-05a6-4897-aae0-85f5ccc21dd0" containerName="revoke-keys" Oct 30 22:38:44.073044 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:44.073037 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03b0d538-05a6-4897-aae0-85f5ccc21dd0" containerName="revoke-keys" Oct 30 22:38:44.073234 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:44.073133 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="03b0d538-05a6-4897-aae0-85f5ccc21dd0" containerName="revoke-keys" Oct 30 22:38:44.076625 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:44.076605 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-506964-pkw4b" Oct 30 22:38:44.132185 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:44.132154 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-506964-pkw4b"] Oct 30 22:38:44.181328 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:44.181291 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8bb708-01af-49f0-b8de-81e0217fcee7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-506964-pkw4b\" (UID: \"0f8bb708-01af-49f0-b8de-81e0217fcee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-506964-pkw4b" Oct 30 22:38:44.181502 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:44.181333 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8bb708-01af-49f0-b8de-81e0217fcee7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-506964-pkw4b\" (UID: \"0f8bb708-01af-49f0-b8de-81e0217fcee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-506964-pkw4b" Oct 30 22:38:44.181502 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:44.181413 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf8mb\" (UniqueName: \"kubernetes.io/projected/0f8bb708-01af-49f0-b8de-81e0217fcee7-kube-api-access-kf8mb\") pod \"maas-api-revoke-keys-e2e-inf-b-506964-pkw4b\" (UID: \"0f8bb708-01af-49f0-b8de-81e0217fcee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-506964-pkw4b" Oct 30 22:38:44.282724 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:44.282680 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8bb708-01af-49f0-b8de-81e0217fcee7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-506964-pkw4b\" (UID: \"0f8bb708-01af-49f0-b8de-81e0217fcee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-506964-pkw4b" Oct 30 22:38:44.282724 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:44.282730 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8bb708-01af-49f0-b8de-81e0217fcee7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-506964-pkw4b\" (UID: \"0f8bb708-01af-49f0-b8de-81e0217fcee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-506964-pkw4b" Oct 30 22:38:44.282999 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:44.282920 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf8mb\" (UniqueName: \"kubernetes.io/projected/0f8bb708-01af-49f0-b8de-81e0217fcee7-kube-api-access-kf8mb\") pod \"maas-api-revoke-keys-e2e-inf-b-506964-pkw4b\" (UID: \"0f8bb708-01af-49f0-b8de-81e0217fcee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-506964-pkw4b" Oct 30 22:38:44.283462 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:44.283440 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8bb708-01af-49f0-b8de-81e0217fcee7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-506964-pkw4b\" (UID: \"0f8bb708-01af-49f0-b8de-81e0217fcee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-506964-pkw4b" Oct 30 22:38:44.285204 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:44.285181 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8bb708-01af-49f0-b8de-81e0217fcee7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-506964-pkw4b\" (UID: \"0f8bb708-01af-49f0-b8de-81e0217fcee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-506964-pkw4b" Oct 30 22:38:44.341526 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:44.341449 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf8mb\" (UniqueName: \"kubernetes.io/projected/0f8bb708-01af-49f0-b8de-81e0217fcee7-kube-api-access-kf8mb\") pod \"maas-api-revoke-keys-e2e-inf-b-506964-pkw4b\" (UID: \"0f8bb708-01af-49f0-b8de-81e0217fcee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-506964-pkw4b" Oct 30 22:38:44.390316 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:44.390276 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-506964-pkw4b" Oct 30 22:38:44.432058 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:44.432023 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32401d4a-6xph4" event={"ID":"03b0d538-05a6-4897-aae0-85f5ccc21dd0","Type":"ContainerDied","Data":"fc3354933d6543740bee84cb1d02ebf962ad21c0dc832051713b7f89ef1308e9"} Oct 30 22:38:44.432235 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:44.432074 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc3354933d6543740bee84cb1d02ebf962ad21c0dc832051713b7f89ef1308e9" Oct 30 22:38:44.432235 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:44.432139 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32401d4a-6xph4" Oct 30 22:38:44.766364 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:44.766339 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-506964-pkw4b"] Oct 30 22:38:44.768366 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:38:44.768332 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8bb708_01af_49f0_b8de_81e0217fcee7.slice/crio-3e8219105534466ae729d1bc2fc6b9178d126e7d3ac7178527d078b4206be22b WatchSource:0}: Error finding container 3e8219105534466ae729d1bc2fc6b9178d126e7d3ac7178527d078b4206be22b: Status 404 returned error can't find the container with id 3e8219105534466ae729d1bc2fc6b9178d126e7d3ac7178527d078b4206be22b Oct 30 22:38:45.440763 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:45.440717 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-506964-pkw4b" event={"ID":"0f8bb708-01af-49f0-b8de-81e0217fcee7","Type":"ContainerStarted","Data":"473399c58f79774683ea2fba30265ade1cc53c52414f43b7d315b9ac515576af"} Oct 30 22:38:45.440943 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:45.440791 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-506964-pkw4b" event={"ID":"0f8bb708-01af-49f0-b8de-81e0217fcee7","Type":"ContainerStarted","Data":"3e8219105534466ae729d1bc2fc6b9178d126e7d3ac7178527d078b4206be22b"} Oct 30 22:38:45.552203 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:45.552133 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-506964-pkw4b" podStartSLOduration=1.5521117420000001 podStartE2EDuration="1.552111742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:45.550411559 +0000 UTC m=+2928.102587859" watchObservedRunningTime="2026-10-30 22:38:45.552111742 +0000 UTC m=+2928.104288042" Oct 30 22:38:46.448425 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:46.448390 6006 generic.go:356] "Generic (PLEG): container finished" podID="0f8bb708-01af-49f0-b8de-81e0217fcee7" containerID="473399c58f79774683ea2fba30265ade1cc53c52414f43b7d315b9ac515576af" exitCode=0 Oct 30 22:38:46.448851 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:46.448482 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-506964-pkw4b" event={"ID":"0f8bb708-01af-49f0-b8de-81e0217fcee7","Type":"ContainerDied","Data":"473399c58f79774683ea2fba30265ade1cc53c52414f43b7d315b9ac515576af"} Oct 30 22:38:47.634471 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:47.634420 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-506964-pkw4b" Oct 30 22:38:47.723184 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:47.723096 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f8bb708-01af-49f0-b8de-81e0217fcee7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8bb708-01af-49f0-b8de-81e0217fcee7-maas-api-openshift-ca\") pod \"0f8bb708-01af-49f0-b8de-81e0217fcee7\" (UID: \"0f8bb708-01af-49f0-b8de-81e0217fcee7\") " Oct 30 22:38:47.723411 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:47.723197 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f8bb708-01af-49f0-b8de-81e0217fcee7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8bb708-01af-49f0-b8de-81e0217fcee7-maas-api-xks-ca\") pod \"0f8bb708-01af-49f0-b8de-81e0217fcee7\" (UID: \"0f8bb708-01af-49f0-b8de-81e0217fcee7\") " Oct 30 22:38:47.723411 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:47.723242 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f8bb708-01af-49f0-b8de-81e0217fcee7-kube-api-access-kf8mb\" (UniqueName: \"kubernetes.io/projected/0f8bb708-01af-49f0-b8de-81e0217fcee7-kube-api-access-kf8mb\") pod \"0f8bb708-01af-49f0-b8de-81e0217fcee7\" (UID: \"0f8bb708-01af-49f0-b8de-81e0217fcee7\") " Oct 30 22:38:47.723563 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:47.723531 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f8bb708-01af-49f0-b8de-81e0217fcee7-maas-api-openshift-ca" pod "0f8bb708-01af-49f0-b8de-81e0217fcee7" (UID: "0f8bb708-01af-49f0-b8de-81e0217fcee7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:38:47.724404 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:47.724096 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8bb708-01af-49f0-b8de-81e0217fcee7-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:47.725727 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:47.725680 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8bb708-01af-49f0-b8de-81e0217fcee7-kube-api-access-kf8mb" pod "0f8bb708-01af-49f0-b8de-81e0217fcee7" (UID: "0f8bb708-01af-49f0-b8de-81e0217fcee7"). InnerVolumeSpecName "kube-api-access-kf8mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:47.726834 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:47.726810 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f8bb708-01af-49f0-b8de-81e0217fcee7-maas-api-xks-ca" pod "0f8bb708-01af-49f0-b8de-81e0217fcee7" (UID: "0f8bb708-01af-49f0-b8de-81e0217fcee7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:47.825469 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:47.825427 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8bb708-01af-49f0-b8de-81e0217fcee7-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:47.825469 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:47.825468 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf8mb\" (UniqueName: \"kubernetes.io/projected/0f8bb708-01af-49f0-b8de-81e0217fcee7-kube-api-access-kf8mb\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:48.462694 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:48.462647 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-506964-pkw4b" event={"ID":"0f8bb708-01af-49f0-b8de-81e0217fcee7","Type":"ContainerDied","Data":"3e8219105534466ae729d1bc2fc6b9178d126e7d3ac7178527d078b4206be22b"} Oct 30 22:38:48.462922 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:48.462711 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e8219105534466ae729d1bc2fc6b9178d126e7d3ac7178527d078b4206be22b" Oct 30 22:38:48.462922 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:48.462667 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-506964-pkw4b" Oct 30 22:38:54.439445 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:54.439406 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" podUID="772ecd37-9f05-48de-91e3-0ba670519f00" containerName="main" containerID="cri-o://20a7715a7d12cf42257e7cf60160ed7c1709b739684198b6257439bade4338df" gracePeriod=45 Oct 30 22:38:54.609663 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:54.609620 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" podUID="772ecd37-9f05-48de-91e3-0ba670519f00" containerName="main" probeResult="failure" output="Get \"https://10.132.0.177:8000/ready\": dial tcp 10.132.0.177:8000: connect: connection refused" Oct 30 22:38:54.739430 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:54.739399 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" Oct 30 22:38:54.903127 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:54.903096 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/772ecd37-9f05-48de-91e3-0ba670519f00-kube-api-access-kfpk5\" (UniqueName: \"kubernetes.io/projected/772ecd37-9f05-48de-91e3-0ba670519f00-kube-api-access-kfpk5\") pod \"772ecd37-9f05-48de-91e3-0ba670519f00\" (UID: \"772ecd37-9f05-48de-91e3-0ba670519f00\") " Oct 30 22:38:54.903337 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:54.903173 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/772ecd37-9f05-48de-91e3-0ba670519f00-tls-certs\" (UniqueName: \"kubernetes.io/secret/772ecd37-9f05-48de-91e3-0ba670519f00-tls-certs\") pod \"772ecd37-9f05-48de-91e3-0ba670519f00\" (UID: \"772ecd37-9f05-48de-91e3-0ba670519f00\") " Oct 30 22:38:54.903337 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:54.903219 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-dshm\" (UniqueName: \"kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-dshm\") pod \"772ecd37-9f05-48de-91e3-0ba670519f00\" (UID: \"772ecd37-9f05-48de-91e3-0ba670519f00\") " Oct 30 22:38:54.903337 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:54.903315 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-tmp-dir\") pod \"772ecd37-9f05-48de-91e3-0ba670519f00\" (UID: \"772ecd37-9f05-48de-91e3-0ba670519f00\") " Oct 30 22:38:54.903541 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:54.903368 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-home\" (UniqueName: \"kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-home\") pod \"772ecd37-9f05-48de-91e3-0ba670519f00\" (UID: \"772ecd37-9f05-48de-91e3-0ba670519f00\") " Oct 30 22:38:54.903541 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:54.903463 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-model-cache\") pod \"772ecd37-9f05-48de-91e3-0ba670519f00\" (UID: \"772ecd37-9f05-48de-91e3-0ba670519f00\") " Oct 30 22:38:54.904373 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:54.903854 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-tmp-dir" pod "772ecd37-9f05-48de-91e3-0ba670519f00" (UID: "772ecd37-9f05-48de-91e3-0ba670519f00"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:54.904373 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:54.903878 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-home" pod "772ecd37-9f05-48de-91e3-0ba670519f00" (UID: "772ecd37-9f05-48de-91e3-0ba670519f00"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:54.904373 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:54.904009 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-model-cache" pod "772ecd37-9f05-48de-91e3-0ba670519f00" (UID: "772ecd37-9f05-48de-91e3-0ba670519f00"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:54.904373 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:54.904313 6006 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-tmp-dir\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:54.904373 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:54.904333 6006 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-home\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:54.904373 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:54.904348 6006 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-model-cache\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:54.906284 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:54.906227 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-dshm" pod "772ecd37-9f05-48de-91e3-0ba670519f00" (UID: "772ecd37-9f05-48de-91e3-0ba670519f00"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:54.906396 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:54.906348 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772ecd37-9f05-48de-91e3-0ba670519f00-kube-api-access-kfpk5" pod "772ecd37-9f05-48de-91e3-0ba670519f00" (UID: "772ecd37-9f05-48de-91e3-0ba670519f00"). InnerVolumeSpecName "kube-api-access-kfpk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:54.906471 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:54.906440 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772ecd37-9f05-48de-91e3-0ba670519f00-tls-certs" pod "772ecd37-9f05-48de-91e3-0ba670519f00" (UID: "772ecd37-9f05-48de-91e3-0ba670519f00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:55.013330 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:55.010380 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfpk5\" (UniqueName: \"kubernetes.io/projected/772ecd37-9f05-48de-91e3-0ba670519f00-kube-api-access-kfpk5\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:55.013330 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:55.010453 6006 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772ecd37-9f05-48de-91e3-0ba670519f00-tls-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:55.013330 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:55.010488 6006 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772ecd37-9f05-48de-91e3-0ba670519f00-dshm\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:55.527139 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:55.527099 6006 generic.go:356] "Generic (PLEG): container finished" podID="772ecd37-9f05-48de-91e3-0ba670519f00" containerID="20a7715a7d12cf42257e7cf60160ed7c1709b739684198b6257439bade4338df" exitCode=0 Oct 30 22:38:55.527695 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:55.527168 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" Oct 30 22:38:55.527695 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:55.527179 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" event={"ID":"772ecd37-9f05-48de-91e3-0ba670519f00","Type":"ContainerDied","Data":"20a7715a7d12cf42257e7cf60160ed7c1709b739684198b6257439bade4338df"} Oct 30 22:38:55.527695 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:55.527233 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz" event={"ID":"772ecd37-9f05-48de-91e3-0ba670519f00","Type":"ContainerDied","Data":"c2d7417e5c12203915471c5006234ba1795a81407eb079acae1de1c914570e5d"} Oct 30 22:38:55.527695 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:55.527318 6006 scope.go:122] "RemoveContainer" containerID="20a7715a7d12cf42257e7cf60160ed7c1709b739684198b6257439bade4338df" Oct 30 22:38:55.540528 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:55.540504 6006 scope.go:122] "RemoveContainer" containerID="20a7715a7d12cf42257e7cf60160ed7c1709b739684198b6257439bade4338df" Oct 30 22:38:55.540853 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:38:55.540829 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20a7715a7d12cf42257e7cf60160ed7c1709b739684198b6257439bade4338df\": container with ID starting with 20a7715a7d12cf42257e7cf60160ed7c1709b739684198b6257439bade4338df not found: ID does not exist" containerID="20a7715a7d12cf42257e7cf60160ed7c1709b739684198b6257439bade4338df" Oct 30 22:38:55.540923 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:55.540865 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20a7715a7d12cf42257e7cf60160ed7c1709b739684198b6257439bade4338df"} err="failed to get container status \"20a7715a7d12cf42257e7cf60160ed7c1709b739684198b6257439bade4338df\": rpc error: code = NotFound desc = could not find container \"20a7715a7d12cf42257e7cf60160ed7c1709b739684198b6257439bade4338df\": container with ID starting with 20a7715a7d12cf42257e7cf60160ed7c1709b739684198b6257439bade4338df not found: ID does not exist" Oct 30 22:38:55.690984 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:55.690947 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz"] Oct 30 22:38:55.731621 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:55.731583 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-75fb62/test-model-75fb62-kserve-b4cb96d67-frtsz"] Oct 30 22:38:57.482711 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:57.482675 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:38:57.603385 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:57.603349 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="772ecd37-9f05-48de-91e3-0ba670519f00" path="/var/lib/kubelet/pods/772ecd37-9f05-48de-91e3-0ba670519f00/volumes" Oct 30 22:38:57.916606 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:38:57.916539 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:38:59.690357 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:38:59.690310 6006 scope.go:122] "RemoveContainer" containerID="5c57e4e8197fc693d6b6de3d29c1f2fd419afac4ee4b296ff63160d46e5e6149" Oct 30 22:39:06.975785 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:06.975744 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj"] Oct 30 22:39:06.976245 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:06.976207 6006 cpu_manager.go:414] "containerMap: removing container" podUID="0f8bb708-01af-49f0-b8de-81e0217fcee7" containerName="revoke-keys" Oct 30 22:39:06.976245 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:06.976219 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f8bb708-01af-49f0-b8de-81e0217fcee7" containerName="revoke-keys" Oct 30 22:39:06.976245 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:06.976242 6006 cpu_manager.go:414] "containerMap: removing container" podUID="772ecd37-9f05-48de-91e3-0ba670519f00" containerName="main" Oct 30 22:39:06.976245 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:06.976274 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="772ecd37-9f05-48de-91e3-0ba670519f00" containerName="main" Oct 30 22:39:06.976516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:06.976342 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f8bb708-01af-49f0-b8de-81e0217fcee7" containerName="revoke-keys" Oct 30 22:39:06.976516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:06.976354 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="772ecd37-9f05-48de-91e3-0ba670519f00" containerName="main" Oct 30 22:39:06.981379 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:06.981356 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj" Oct 30 22:39:07.055188 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:07.055150 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxn7k\" (UniqueName: \"kubernetes.io/projected/41db76b4-54ca-47b2-aed2-4ba3e03f1e7a-kube-api-access-jxn7k\") pod \"maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj\" (UID: \"41db76b4-54ca-47b2-aed2-4ba3e03f1e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj" Oct 30 22:39:07.055416 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:07.055211 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41db76b4-54ca-47b2-aed2-4ba3e03f1e7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj\" (UID: \"41db76b4-54ca-47b2-aed2-4ba3e03f1e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj" Oct 30 22:39:07.055416 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:07.055351 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41db76b4-54ca-47b2-aed2-4ba3e03f1e7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj\" (UID: \"41db76b4-54ca-47b2-aed2-4ba3e03f1e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj" Oct 30 22:39:07.079389 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:07.079348 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:39:07.079582 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:07.079430 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:39:07.079582 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:07.079426 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:39:07.079582 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:07.079475 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:39:07.081591 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:07.081564 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj"] Oct 30 22:39:07.156793 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:07.156755 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41db76b4-54ca-47b2-aed2-4ba3e03f1e7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj\" (UID: \"41db76b4-54ca-47b2-aed2-4ba3e03f1e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj" Oct 30 22:39:07.156988 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:07.156912 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41db76b4-54ca-47b2-aed2-4ba3e03f1e7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj\" (UID: \"41db76b4-54ca-47b2-aed2-4ba3e03f1e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj" Oct 30 22:39:07.157059 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:07.157043 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxn7k\" (UniqueName: \"kubernetes.io/projected/41db76b4-54ca-47b2-aed2-4ba3e03f1e7a-kube-api-access-jxn7k\") pod \"maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj\" (UID: \"41db76b4-54ca-47b2-aed2-4ba3e03f1e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj" Oct 30 22:39:07.157483 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:07.157460 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41db76b4-54ca-47b2-aed2-4ba3e03f1e7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj\" (UID: \"41db76b4-54ca-47b2-aed2-4ba3e03f1e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj" Oct 30 22:39:07.159303 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:07.159281 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41db76b4-54ca-47b2-aed2-4ba3e03f1e7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj\" (UID: \"41db76b4-54ca-47b2-aed2-4ba3e03f1e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj" Oct 30 22:39:07.214987 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:07.214949 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxn7k\" (UniqueName: \"kubernetes.io/projected/41db76b4-54ca-47b2-aed2-4ba3e03f1e7a-kube-api-access-jxn7k\") pod \"maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj\" (UID: \"41db76b4-54ca-47b2-aed2-4ba3e03f1e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj" Oct 30 22:39:07.303082 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:07.303047 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj" Oct 30 22:39:07.677009 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:07.676986 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj"] Oct 30 22:39:07.680041 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:39:07.680007 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41db76b4_54ca_47b2_aed2_4ba3e03f1e7a.slice/crio-d50583a91774f78f09542ad53d843ae7625e69bd5bdb6521e82e452ddaeceb69 WatchSource:0}: Error finding container d50583a91774f78f09542ad53d843ae7625e69bd5bdb6521e82e452ddaeceb69: Status 404 returned error can't find the container with id d50583a91774f78f09542ad53d843ae7625e69bd5bdb6521e82e452ddaeceb69 Oct 30 22:39:08.619831 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:08.619790 6006 generic.go:356] "Generic (PLEG): container finished" podID="41db76b4-54ca-47b2-aed2-4ba3e03f1e7a" containerID="9c4092cd72e1af63cfaed71a49cdc1c1ed718b8cf89632a1c503dd433e00b504" exitCode=0 Oct 30 22:39:08.620344 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:08.619874 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj" event={"ID":"41db76b4-54ca-47b2-aed2-4ba3e03f1e7a","Type":"ContainerDied","Data":"9c4092cd72e1af63cfaed71a49cdc1c1ed718b8cf89632a1c503dd433e00b504"} Oct 30 22:39:08.620344 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:08.619929 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj" event={"ID":"41db76b4-54ca-47b2-aed2-4ba3e03f1e7a","Type":"ContainerStarted","Data":"d50583a91774f78f09542ad53d843ae7625e69bd5bdb6521e82e452ddaeceb69"} Oct 30 22:39:09.779405 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:09.779379 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj" Oct 30 22:39:09.909579 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:09.909481 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41db76b4-54ca-47b2-aed2-4ba3e03f1e7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41db76b4-54ca-47b2-aed2-4ba3e03f1e7a-maas-api-openshift-ca\") pod \"41db76b4-54ca-47b2-aed2-4ba3e03f1e7a\" (UID: \"41db76b4-54ca-47b2-aed2-4ba3e03f1e7a\") " Oct 30 22:39:09.909579 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:09.909562 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41db76b4-54ca-47b2-aed2-4ba3e03f1e7a-kube-api-access-jxn7k\" (UniqueName: \"kubernetes.io/projected/41db76b4-54ca-47b2-aed2-4ba3e03f1e7a-kube-api-access-jxn7k\") pod \"41db76b4-54ca-47b2-aed2-4ba3e03f1e7a\" (UID: \"41db76b4-54ca-47b2-aed2-4ba3e03f1e7a\") " Oct 30 22:39:09.909818 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:09.909712 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41db76b4-54ca-47b2-aed2-4ba3e03f1e7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41db76b4-54ca-47b2-aed2-4ba3e03f1e7a-maas-api-xks-ca\") pod \"41db76b4-54ca-47b2-aed2-4ba3e03f1e7a\" (UID: \"41db76b4-54ca-47b2-aed2-4ba3e03f1e7a\") " Oct 30 22:39:09.909986 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:09.909957 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41db76b4-54ca-47b2-aed2-4ba3e03f1e7a-maas-api-openshift-ca" pod "41db76b4-54ca-47b2-aed2-4ba3e03f1e7a" (UID: "41db76b4-54ca-47b2-aed2-4ba3e03f1e7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:39:09.910095 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:09.910074 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41db76b4-54ca-47b2-aed2-4ba3e03f1e7a-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:09.912000 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:09.911968 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41db76b4-54ca-47b2-aed2-4ba3e03f1e7a-maas-api-xks-ca" pod "41db76b4-54ca-47b2-aed2-4ba3e03f1e7a" (UID: "41db76b4-54ca-47b2-aed2-4ba3e03f1e7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:39:09.912231 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:09.912212 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41db76b4-54ca-47b2-aed2-4ba3e03f1e7a-kube-api-access-jxn7k" pod "41db76b4-54ca-47b2-aed2-4ba3e03f1e7a" (UID: "41db76b4-54ca-47b2-aed2-4ba3e03f1e7a"). InnerVolumeSpecName "kube-api-access-jxn7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:39:10.011529 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:10.011487 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41db76b4-54ca-47b2-aed2-4ba3e03f1e7a-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:10.011529 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:10.011522 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxn7k\" (UniqueName: \"kubernetes.io/projected/41db76b4-54ca-47b2-aed2-4ba3e03f1e7a-kube-api-access-jxn7k\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:10.634918 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:10.634857 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj" Oct 30 22:39:10.634918 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:10.634880 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj" event={"ID":"41db76b4-54ca-47b2-aed2-4ba3e03f1e7a","Type":"ContainerDied","Data":"d50583a91774f78f09542ad53d843ae7625e69bd5bdb6521e82e452ddaeceb69"} Oct 30 22:39:10.635152 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:10.634941 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d50583a91774f78f09542ad53d843ae7625e69bd5bdb6521e82e452ddaeceb69" Oct 30 22:39:24.529380 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:24.529340 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-506964-pkw4b"] Oct 30 22:39:24.575791 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:24.575753 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-506964-pkw4b"] Oct 30 22:39:25.604391 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:25.604353 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f8bb708-01af-49f0-b8de-81e0217fcee7" path="/var/lib/kubelet/pods/0f8bb708-01af-49f0-b8de-81e0217fcee7/volumes" Oct 30 22:39:27.535601 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:27.535560 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:39:27.911449 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:39:27.911411 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:39:32.460809 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:32.460768 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg"] Oct 30 22:39:32.461225 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:32.461212 6006 cpu_manager.go:414] "containerMap: removing container" podUID="41db76b4-54ca-47b2-aed2-4ba3e03f1e7a" containerName="revoke-keys" Oct 30 22:39:32.461301 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:32.461225 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41db76b4-54ca-47b2-aed2-4ba3e03f1e7a" containerName="revoke-keys" Oct 30 22:39:32.461347 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:32.461340 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="41db76b4-54ca-47b2-aed2-4ba3e03f1e7a" containerName="revoke-keys" Oct 30 22:39:32.465023 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:32.464994 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg" Oct 30 22:39:32.517198 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:32.517155 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:39:32.517486 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:32.517455 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:39:32.545062 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:32.545020 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlccw\" (UniqueName: \"kubernetes.io/projected/6ebe5ecb-b0ce-450b-965d-d7c768996307-kube-api-access-rlccw\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg\" (UID: \"6ebe5ecb-b0ce-450b-965d-d7c768996307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg" Oct 30 22:39:32.545244 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:32.545076 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebe5ecb-b0ce-450b-965d-d7c768996307-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg\" (UID: \"6ebe5ecb-b0ce-450b-965d-d7c768996307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg" Oct 30 22:39:32.545244 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:32.545160 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebe5ecb-b0ce-450b-965d-d7c768996307-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg\" (UID: \"6ebe5ecb-b0ce-450b-965d-d7c768996307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg" Oct 30 22:39:32.563951 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:32.563914 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:39:32.563951 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:32.563945 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:39:32.566083 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:32.566060 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg"] Oct 30 22:39:32.646601 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:32.646564 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlccw\" (UniqueName: \"kubernetes.io/projected/6ebe5ecb-b0ce-450b-965d-d7c768996307-kube-api-access-rlccw\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg\" (UID: \"6ebe5ecb-b0ce-450b-965d-d7c768996307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg" Oct 30 22:39:32.646820 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:32.646637 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebe5ecb-b0ce-450b-965d-d7c768996307-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg\" (UID: \"6ebe5ecb-b0ce-450b-965d-d7c768996307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg" Oct 30 22:39:32.646820 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:32.646666 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebe5ecb-b0ce-450b-965d-d7c768996307-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg\" (UID: \"6ebe5ecb-b0ce-450b-965d-d7c768996307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg" Oct 30 22:39:32.647579 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:32.647554 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebe5ecb-b0ce-450b-965d-d7c768996307-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg\" (UID: \"6ebe5ecb-b0ce-450b-965d-d7c768996307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg" Oct 30 22:39:32.649316 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:32.649289 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebe5ecb-b0ce-450b-965d-d7c768996307-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg\" (UID: \"6ebe5ecb-b0ce-450b-965d-d7c768996307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg" Oct 30 22:39:32.704791 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:32.704759 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlccw\" (UniqueName: \"kubernetes.io/projected/6ebe5ecb-b0ce-450b-965d-d7c768996307-kube-api-access-rlccw\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg\" (UID: \"6ebe5ecb-b0ce-450b-965d-d7c768996307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg" Oct 30 22:39:32.778465 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:32.778378 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg" Oct 30 22:39:33.151663 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:33.151633 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg"] Oct 30 22:39:33.154430 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:39:33.154383 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebe5ecb_b0ce_450b_965d_d7c768996307.slice/crio-fe282a1b0c8f117957a872c873d1a19985719a36cbbc9d83bafe96237c8e2a77 WatchSource:0}: Error finding container fe282a1b0c8f117957a872c873d1a19985719a36cbbc9d83bafe96237c8e2a77: Status 404 returned error can't find the container with id fe282a1b0c8f117957a872c873d1a19985719a36cbbc9d83bafe96237c8e2a77 Oct 30 22:39:33.780824 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:33.780730 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg" event={"ID":"6ebe5ecb-b0ce-450b-965d-d7c768996307","Type":"ContainerStarted","Data":"2801ab17f7c3e15202450d85e720c3f913996f6b2ff72ad5cfc965bd99bff439"} Oct 30 22:39:33.780824 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:33.780791 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg" event={"ID":"6ebe5ecb-b0ce-450b-965d-d7c768996307","Type":"ContainerStarted","Data":"fe282a1b0c8f117957a872c873d1a19985719a36cbbc9d83bafe96237c8e2a77"} Oct 30 22:39:33.891502 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:33.891443 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg" podStartSLOduration=1.8914276559999998 podStartE2EDuration="1.891427656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:33.890194184 +0000 UTC m=+2976.442370484" watchObservedRunningTime="2026-10-30 22:39:33.891427656 +0000 UTC m=+2976.443603955" Oct 30 22:39:34.787818 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:34.787782 6006 generic.go:356] "Generic (PLEG): container finished" podID="6ebe5ecb-b0ce-450b-965d-d7c768996307" containerID="2801ab17f7c3e15202450d85e720c3f913996f6b2ff72ad5cfc965bd99bff439" exitCode=0 Oct 30 22:39:34.788341 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:34.787875 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg" event={"ID":"6ebe5ecb-b0ce-450b-965d-d7c768996307","Type":"ContainerDied","Data":"2801ab17f7c3e15202450d85e720c3f913996f6b2ff72ad5cfc965bd99bff439"} Oct 30 22:39:35.460777 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:35.460738 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj"] Oct 30 22:39:35.506458 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:35.506417 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-75fb62-hb8hj"] Oct 30 22:39:35.603203 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:35.603162 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41db76b4-54ca-47b2-aed2-4ba3e03f1e7a" path="/var/lib/kubelet/pods/41db76b4-54ca-47b2-aed2-4ba3e03f1e7a/volumes" Oct 30 22:39:35.951948 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:35.951917 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg" Oct 30 22:39:35.987079 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:35.987042 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ebe5ecb-b0ce-450b-965d-d7c768996307-kube-api-access-rlccw\" (UniqueName: \"kubernetes.io/projected/6ebe5ecb-b0ce-450b-965d-d7c768996307-kube-api-access-rlccw\") pod \"6ebe5ecb-b0ce-450b-965d-d7c768996307\" (UID: \"6ebe5ecb-b0ce-450b-965d-d7c768996307\") " Oct 30 22:39:35.987321 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:35.987207 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ebe5ecb-b0ce-450b-965d-d7c768996307-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebe5ecb-b0ce-450b-965d-d7c768996307-maas-api-xks-ca\") pod \"6ebe5ecb-b0ce-450b-965d-d7c768996307\" (UID: \"6ebe5ecb-b0ce-450b-965d-d7c768996307\") " Oct 30 22:39:35.987321 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:35.987265 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ebe5ecb-b0ce-450b-965d-d7c768996307-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebe5ecb-b0ce-450b-965d-d7c768996307-maas-api-openshift-ca\") pod \"6ebe5ecb-b0ce-450b-965d-d7c768996307\" (UID: \"6ebe5ecb-b0ce-450b-965d-d7c768996307\") " Oct 30 22:39:35.987668 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:35.987644 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ebe5ecb-b0ce-450b-965d-d7c768996307-maas-api-openshift-ca" pod "6ebe5ecb-b0ce-450b-965d-d7c768996307" (UID: "6ebe5ecb-b0ce-450b-965d-d7c768996307"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:39:35.989461 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:35.989417 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebe5ecb-b0ce-450b-965d-d7c768996307-kube-api-access-rlccw" pod "6ebe5ecb-b0ce-450b-965d-d7c768996307" (UID: "6ebe5ecb-b0ce-450b-965d-d7c768996307"). InnerVolumeSpecName "kube-api-access-rlccw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:39:35.989660 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:35.989490 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ebe5ecb-b0ce-450b-965d-d7c768996307-maas-api-xks-ca" pod "6ebe5ecb-b0ce-450b-965d-d7c768996307" (UID: "6ebe5ecb-b0ce-450b-965d-d7c768996307"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:39:36.088729 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:36.088629 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebe5ecb-b0ce-450b-965d-d7c768996307-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:36.088729 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:36.088662 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebe5ecb-b0ce-450b-965d-d7c768996307-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:36.088729 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:36.088673 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlccw\" (UniqueName: \"kubernetes.io/projected/6ebe5ecb-b0ce-450b-965d-d7c768996307-kube-api-access-rlccw\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:36.800452 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:36.800412 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg" event={"ID":"6ebe5ecb-b0ce-450b-965d-d7c768996307","Type":"ContainerDied","Data":"fe282a1b0c8f117957a872c873d1a19985719a36cbbc9d83bafe96237c8e2a77"} Oct 30 22:39:36.800641 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:36.800459 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg" Oct 30 22:39:36.800641 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:36.800465 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe282a1b0c8f117957a872c873d1a19985719a36cbbc9d83bafe96237c8e2a77" Oct 30 22:39:39.888211 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:39.888164 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj"] Oct 30 22:39:39.888816 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:39.888424 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" podUID="b77193b8-0b50-4d53-beb9-619be2ea30b5" containerName="istio-proxy" containerID="cri-o://75d1617e04a28e90cec827062429e342d99d54c6f48e119ade4b7b0665b14c7f" gracePeriod=30 Oct 30 22:39:41.040960 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:41.040921 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:39:42.625103 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:42.625057 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gkxzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:39:42.625749 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:42.625710 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" podUID="b77193b8-0b50-4d53-beb9-619be2ea30b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:39:43.460355 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:43.460311 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4"] Oct 30 22:39:43.461009 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:43.460989 6006 cpu_manager.go:414] "containerMap: removing container" podUID="6ebe5ecb-b0ce-450b-965d-d7c768996307" containerName="revoke-keys" Oct 30 22:39:43.461064 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:43.461013 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ebe5ecb-b0ce-450b-965d-d7c768996307" containerName="revoke-keys" Oct 30 22:39:43.461183 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:43.461171 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ebe5ecb-b0ce-450b-965d-d7c768996307" containerName="revoke-keys" Oct 30 22:39:43.466241 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:43.466216 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4" Oct 30 22:39:43.518532 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:43.518496 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4"] Oct 30 22:39:43.562532 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:43.562496 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:39:43.562532 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:43.562511 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:39:43.562778 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:43.562514 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:39:43.562778 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:43.562593 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:39:43.566989 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:43.566962 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4p27\" (UniqueName: \"kubernetes.io/projected/21b585aa-5c59-491c-bc39-10f6afee58a3-kube-api-access-b4p27\") pod \"maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4\" (UID: \"21b585aa-5c59-491c-bc39-10f6afee58a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4" Oct 30 22:39:43.567163 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:43.566999 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b585aa-5c59-491c-bc39-10f6afee58a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4\" (UID: \"21b585aa-5c59-491c-bc39-10f6afee58a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4" Oct 30 22:39:43.567238 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:43.567208 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b585aa-5c59-491c-bc39-10f6afee58a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4\" (UID: \"21b585aa-5c59-491c-bc39-10f6afee58a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4" Oct 30 22:39:43.667889 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:43.667849 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b585aa-5c59-491c-bc39-10f6afee58a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4\" (UID: \"21b585aa-5c59-491c-bc39-10f6afee58a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4" Oct 30 22:39:43.668472 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:43.667942 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4p27\" (UniqueName: \"kubernetes.io/projected/21b585aa-5c59-491c-bc39-10f6afee58a3-kube-api-access-b4p27\") pod \"maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4\" (UID: \"21b585aa-5c59-491c-bc39-10f6afee58a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4" Oct 30 22:39:43.668472 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:43.667976 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b585aa-5c59-491c-bc39-10f6afee58a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4\" (UID: \"21b585aa-5c59-491c-bc39-10f6afee58a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4" Oct 30 22:39:43.668720 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:43.668696 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b585aa-5c59-491c-bc39-10f6afee58a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4\" (UID: \"21b585aa-5c59-491c-bc39-10f6afee58a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4" Oct 30 22:39:43.670589 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:43.670562 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b585aa-5c59-491c-bc39-10f6afee58a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4\" (UID: \"21b585aa-5c59-491c-bc39-10f6afee58a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4" Oct 30 22:39:43.726237 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:43.726156 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4p27\" (UniqueName: \"kubernetes.io/projected/21b585aa-5c59-491c-bc39-10f6afee58a3-kube-api-access-b4p27\") pod \"maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4\" (UID: \"21b585aa-5c59-491c-bc39-10f6afee58a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4" Oct 30 22:39:43.780240 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:43.780196 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4" Oct 30 22:39:44.151218 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:44.151181 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4"] Oct 30 22:39:44.154936 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:39:44.154906 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b585aa_5c59_491c_bc39_10f6afee58a3.slice/crio-a8a8cc2ec927c9a0a0533ab558c58331d41e759fc3f1a392469fa4cf69d75d36 WatchSource:0}: Error finding container a8a8cc2ec927c9a0a0533ab558c58331d41e759fc3f1a392469fa4cf69d75d36: Status 404 returned error can't find the container with id a8a8cc2ec927c9a0a0533ab558c58331d41e759fc3f1a392469fa4cf69d75d36 Oct 30 22:39:44.861087 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:44.861044 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4" event={"ID":"21b585aa-5c59-491c-bc39-10f6afee58a3","Type":"ContainerStarted","Data":"bae079c7c3bf8302fda12b786f5f98f5f2946cc53cc664c73c1d45834e6461e5"} Oct 30 22:39:44.861565 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:44.861111 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4" event={"ID":"21b585aa-5c59-491c-bc39-10f6afee58a3","Type":"ContainerStarted","Data":"a8a8cc2ec927c9a0a0533ab558c58331d41e759fc3f1a392469fa4cf69d75d36"} Oct 30 22:39:44.970903 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:44.970841 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4" podStartSLOduration=1.970823974 podStartE2EDuration="1.970823974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:44.970231994 +0000 UTC m=+2987.522408297" watchObservedRunningTime="2026-10-30 22:39:44.970823974 +0000 UTC m=+2987.523000276" Oct 30 22:39:45.154717 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.154690 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 22:39:45.287685 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.287648 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-envoy\") pod \"b77193b8-0b50-4d53-beb9-619be2ea30b5\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " Oct 30 22:39:45.287883 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.287713 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b77193b8-0b50-4d53-beb9-619be2ea30b5-kube-api-access-r9jjq\" (UniqueName: \"kubernetes.io/projected/b77193b8-0b50-4d53-beb9-619be2ea30b5-kube-api-access-r9jjq\") pod \"b77193b8-0b50-4d53-beb9-619be2ea30b5\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " Oct 30 22:39:45.287883 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.287770 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-workload-socket\") pod \"b77193b8-0b50-4d53-beb9-619be2ea30b5\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " Oct 30 22:39:45.287883 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.287816 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-token\" (UniqueName: \"kubernetes.io/projected/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-token\") pod \"b77193b8-0b50-4d53-beb9-619be2ea30b5\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " Oct 30 22:39:45.287883 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.287844 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-data\") pod \"b77193b8-0b50-4d53-beb9-619be2ea30b5\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " Oct 30 22:39:45.288136 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.288089 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-workload-socket" pod "b77193b8-0b50-4d53-beb9-619be2ea30b5" (UID: "b77193b8-0b50-4d53-beb9-619be2ea30b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:45.288136 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.288098 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-credential-socket\") pod \"b77193b8-0b50-4d53-beb9-619be2ea30b5\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " Oct 30 22:39:45.288281 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.288103 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-data" pod "b77193b8-0b50-4d53-beb9-619be2ea30b5" (UID: "b77193b8-0b50-4d53-beb9-619be2ea30b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:45.288350 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.288327 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-podinfo\") pod \"b77193b8-0b50-4d53-beb9-619be2ea30b5\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " Oct 30 22:39:45.288407 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.288364 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b77193b8-0b50-4d53-beb9-619be2ea30b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b77193b8-0b50-4d53-beb9-619be2ea30b5-istiod-ca-cert\") pod \"b77193b8-0b50-4d53-beb9-619be2ea30b5\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " Oct 30 22:39:45.288465 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.288424 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-workload-certs\") pod \"b77193b8-0b50-4d53-beb9-619be2ea30b5\" (UID: \"b77193b8-0b50-4d53-beb9-619be2ea30b5\") " Oct 30 22:39:45.288529 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.288487 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-credential-socket" pod "b77193b8-0b50-4d53-beb9-619be2ea30b5" (UID: "b77193b8-0b50-4d53-beb9-619be2ea30b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:45.288851 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.288803 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b77193b8-0b50-4d53-beb9-619be2ea30b5-istiod-ca-cert" pod "b77193b8-0b50-4d53-beb9-619be2ea30b5" (UID: "b77193b8-0b50-4d53-beb9-619be2ea30b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:39:45.288851 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.288813 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:45.288851 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.288850 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:45.289098 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.288868 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:45.289098 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.289024 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-workload-certs" pod "b77193b8-0b50-4d53-beb9-619be2ea30b5" (UID: "b77193b8-0b50-4d53-beb9-619be2ea30b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:45.290335 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.290314 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-envoy" pod "b77193b8-0b50-4d53-beb9-619be2ea30b5" (UID: "b77193b8-0b50-4d53-beb9-619be2ea30b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:45.290593 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.290577 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b77193b8-0b50-4d53-beb9-619be2ea30b5-kube-api-access-r9jjq" pod "b77193b8-0b50-4d53-beb9-619be2ea30b5" (UID: "b77193b8-0b50-4d53-beb9-619be2ea30b5"). InnerVolumeSpecName "kube-api-access-r9jjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:39:45.291001 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.290977 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-podinfo" pod "b77193b8-0b50-4d53-beb9-619be2ea30b5" (UID: "b77193b8-0b50-4d53-beb9-619be2ea30b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:39:45.291179 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.291161 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-token" pod "b77193b8-0b50-4d53-beb9-619be2ea30b5" (UID: "b77193b8-0b50-4d53-beb9-619be2ea30b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:39:45.390315 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.390196 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:45.390315 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.390232 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:45.390315 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.390242 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b77193b8-0b50-4d53-beb9-619be2ea30b5-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:45.390315 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.390282 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:45.390315 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.390292 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b77193b8-0b50-4d53-beb9-619be2ea30b5-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:45.390315 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.390301 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9jjq\" (UniqueName: \"kubernetes.io/projected/b77193b8-0b50-4d53-beb9-619be2ea30b5-kube-api-access-r9jjq\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:45.869597 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.869561 6006 generic.go:356] "Generic (PLEG): container finished" podID="b77193b8-0b50-4d53-beb9-619be2ea30b5" containerID="75d1617e04a28e90cec827062429e342d99d54c6f48e119ade4b7b0665b14c7f" exitCode=0 Oct 30 22:39:45.870040 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.869662 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" event={"ID":"b77193b8-0b50-4d53-beb9-619be2ea30b5","Type":"ContainerDied","Data":"75d1617e04a28e90cec827062429e342d99d54c6f48e119ade4b7b0665b14c7f"} Oct 30 22:39:45.870040 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.869702 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" event={"ID":"b77193b8-0b50-4d53-beb9-619be2ea30b5","Type":"ContainerDied","Data":"36ec34f1409e67be254250a407346c4827d79db422a337097068a07126c823c7"} Oct 30 22:39:45.870040 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.869721 6006 scope.go:122] "RemoveContainer" containerID="75d1617e04a28e90cec827062429e342d99d54c6f48e119ade4b7b0665b14c7f" Oct 30 22:39:45.870040 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.869847 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj" Oct 30 22:39:45.872182 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.872051 6006 generic.go:356] "Generic (PLEG): container finished" podID="21b585aa-5c59-491c-bc39-10f6afee58a3" containerID="bae079c7c3bf8302fda12b786f5f98f5f2946cc53cc664c73c1d45834e6461e5" exitCode=0 Oct 30 22:39:45.872182 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.872112 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4" event={"ID":"21b585aa-5c59-491c-bc39-10f6afee58a3","Type":"ContainerDied","Data":"bae079c7c3bf8302fda12b786f5f98f5f2946cc53cc664c73c1d45834e6461e5"} Oct 30 22:39:45.879914 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.879887 6006 scope.go:122] "RemoveContainer" containerID="1a4656ab03bb4e27ff95a8e1f968c1faaccb8aa82288c28504bc0dfb9d32e167" Oct 30 22:39:45.889507 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.889484 6006 scope.go:122] "RemoveContainer" containerID="75d1617e04a28e90cec827062429e342d99d54c6f48e119ade4b7b0665b14c7f" Oct 30 22:39:45.889812 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:39:45.889788 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d1617e04a28e90cec827062429e342d99d54c6f48e119ade4b7b0665b14c7f\": container with ID starting with 75d1617e04a28e90cec827062429e342d99d54c6f48e119ade4b7b0665b14c7f not found: ID does not exist" containerID="75d1617e04a28e90cec827062429e342d99d54c6f48e119ade4b7b0665b14c7f" Oct 30 22:39:45.889898 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.889826 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d1617e04a28e90cec827062429e342d99d54c6f48e119ade4b7b0665b14c7f"} err="failed to get container status \"75d1617e04a28e90cec827062429e342d99d54c6f48e119ade4b7b0665b14c7f\": rpc error: code = NotFound desc = could not find container \"75d1617e04a28e90cec827062429e342d99d54c6f48e119ade4b7b0665b14c7f\": container with ID starting with 75d1617e04a28e90cec827062429e342d99d54c6f48e119ade4b7b0665b14c7f not found: ID does not exist" Oct 30 22:39:45.889898 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.889852 6006 scope.go:122] "RemoveContainer" containerID="1a4656ab03bb4e27ff95a8e1f968c1faaccb8aa82288c28504bc0dfb9d32e167" Oct 30 22:39:45.890154 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:39:45.890136 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a4656ab03bb4e27ff95a8e1f968c1faaccb8aa82288c28504bc0dfb9d32e167\": container with ID starting with 1a4656ab03bb4e27ff95a8e1f968c1faaccb8aa82288c28504bc0dfb9d32e167 not found: ID does not exist" containerID="1a4656ab03bb4e27ff95a8e1f968c1faaccb8aa82288c28504bc0dfb9d32e167" Oct 30 22:39:45.890191 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:45.890163 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a4656ab03bb4e27ff95a8e1f968c1faaccb8aa82288c28504bc0dfb9d32e167"} err="failed to get container status \"1a4656ab03bb4e27ff95a8e1f968c1faaccb8aa82288c28504bc0dfb9d32e167\": rpc error: code = NotFound desc = could not find container \"1a4656ab03bb4e27ff95a8e1f968c1faaccb8aa82288c28504bc0dfb9d32e167\": container with ID starting with 1a4656ab03bb4e27ff95a8e1f968c1faaccb8aa82288c28504bc0dfb9d32e167 not found: ID does not exist" Oct 30 22:39:46.141640 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:46.141606 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj"] Oct 30 22:39:46.148340 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:46.148307 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gkxzj"] Oct 30 22:39:47.026485 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:47.026454 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4" Oct 30 22:39:47.111034 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:47.110993 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21b585aa-5c59-491c-bc39-10f6afee58a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b585aa-5c59-491c-bc39-10f6afee58a3-maas-api-openshift-ca\") pod \"21b585aa-5c59-491c-bc39-10f6afee58a3\" (UID: \"21b585aa-5c59-491c-bc39-10f6afee58a3\") " Oct 30 22:39:47.111282 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:47.111110 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b585aa-5c59-491c-bc39-10f6afee58a3-kube-api-access-b4p27\" (UniqueName: \"kubernetes.io/projected/21b585aa-5c59-491c-bc39-10f6afee58a3-kube-api-access-b4p27\") pod \"21b585aa-5c59-491c-bc39-10f6afee58a3\" (UID: \"21b585aa-5c59-491c-bc39-10f6afee58a3\") " Oct 30 22:39:47.111282 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:47.111157 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21b585aa-5c59-491c-bc39-10f6afee58a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b585aa-5c59-491c-bc39-10f6afee58a3-maas-api-xks-ca\") pod \"21b585aa-5c59-491c-bc39-10f6afee58a3\" (UID: \"21b585aa-5c59-491c-bc39-10f6afee58a3\") " Oct 30 22:39:47.111442 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:47.111415 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b585aa-5c59-491c-bc39-10f6afee58a3-maas-api-openshift-ca" pod "21b585aa-5c59-491c-bc39-10f6afee58a3" (UID: "21b585aa-5c59-491c-bc39-10f6afee58a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:39:47.113382 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:47.113353 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b585aa-5c59-491c-bc39-10f6afee58a3-maas-api-xks-ca" pod "21b585aa-5c59-491c-bc39-10f6afee58a3" (UID: "21b585aa-5c59-491c-bc39-10f6afee58a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:39:47.113485 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:47.113398 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b585aa-5c59-491c-bc39-10f6afee58a3-kube-api-access-b4p27" pod "21b585aa-5c59-491c-bc39-10f6afee58a3" (UID: "21b585aa-5c59-491c-bc39-10f6afee58a3"). InnerVolumeSpecName "kube-api-access-b4p27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:39:47.213077 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:47.212977 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4p27\" (UniqueName: \"kubernetes.io/projected/21b585aa-5c59-491c-bc39-10f6afee58a3-kube-api-access-b4p27\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:47.213077 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:47.213027 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b585aa-5c59-491c-bc39-10f6afee58a3-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:47.213077 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:47.213042 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b585aa-5c59-491c-bc39-10f6afee58a3-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:47.604513 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:47.604469 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b77193b8-0b50-4d53-beb9-619be2ea30b5" path="/var/lib/kubelet/pods/b77193b8-0b50-4d53-beb9-619be2ea30b5/volumes" Oct 30 22:39:47.886582 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:47.886496 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4" Oct 30 22:39:47.886582 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:47.886522 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4" event={"ID":"21b585aa-5c59-491c-bc39-10f6afee58a3","Type":"ContainerDied","Data":"a8a8cc2ec927c9a0a0533ab558c58331d41e759fc3f1a392469fa4cf69d75d36"} Oct 30 22:39:47.886582 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:47.886568 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8a8cc2ec927c9a0a0533ab558c58331d41e759fc3f1a392469fa4cf69d75d36" Oct 30 22:39:56.093554 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:56.093514 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg"] Oct 30 22:39:56.140279 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:56.140212 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a6b6a5-2v6hg"] Oct 30 22:39:57.603397 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:57.603364 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ebe5ecb-b0ce-450b-965d-d7c768996307" path="/var/lib/kubelet/pods/6ebe5ecb-b0ce-450b-965d-d7c768996307/volumes" Oct 30 22:39:57.910985 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:39:57.910890 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:39:58.144056 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:58.144001 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 22:39:58.162855 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:58.162779 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 22:39:59.723208 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:59.723169 6006 scope.go:122] "RemoveContainer" containerID="c98923e0aa69c3884248595b8dd57921fc2f9d91e261419e376acaba078eec78" Oct 30 22:39:59.732632 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:59.732606 6006 scope.go:122] "RemoveContainer" containerID="6c1fd63db5c7991fd246b0538604d221ef96f2a65ae743ade7fac65008ef7ece" Oct 30 22:39:59.742074 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:39:59.742047 6006 scope.go:122] "RemoveContainer" containerID="732718fcd2aade34aebb8e1aee8c287291712619cf6eac064ac1e6306ddbdfce" Oct 30 22:40:02.167745 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:02.167707 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4"] Oct 30 22:40:02.214070 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:02.214030 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-07e9c8-zl7t4"] Oct 30 22:40:03.048661 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:03.048620 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee281591-d6ztp"] Oct 30 22:40:03.094179 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:03.094140 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b597e2f3-qbxfb"] Oct 30 22:40:03.095768 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:03.095737 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee281591-d6ztp"] Oct 30 22:40:03.097128 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:03.097103 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b597e2f3-qbxfb"] Oct 30 22:40:03.602669 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:03.602626 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2011ac7e-5d4d-4368-8a5a-55cdd5de789e" path="/var/lib/kubelet/pods/2011ac7e-5d4d-4368-8a5a-55cdd5de789e/volumes" Oct 30 22:40:03.603265 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:03.603227 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21b585aa-5c59-491c-bc39-10f6afee58a3" path="/var/lib/kubelet/pods/21b585aa-5c59-491c-bc39-10f6afee58a3/volumes" Oct 30 22:40:03.603700 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:03.603679 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4660bcf0-9e67-41c3-b480-c6b25dfbc9b9" path="/var/lib/kubelet/pods/4660bcf0-9e67-41c3-b480-c6b25dfbc9b9/volumes" Oct 30 22:40:04.777162 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:04.777126 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg"] Oct 30 22:40:04.777589 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:04.777569 6006 cpu_manager.go:414] "containerMap: removing container" podUID="21b585aa-5c59-491c-bc39-10f6afee58a3" containerName="revoke-keys" Oct 30 22:40:04.777589 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:04.777580 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21b585aa-5c59-491c-bc39-10f6afee58a3" containerName="revoke-keys" Oct 30 22:40:04.777668 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:04.777592 6006 cpu_manager.go:414] "containerMap: removing container" podUID="b77193b8-0b50-4d53-beb9-619be2ea30b5" containerName="istio-proxy" Oct 30 22:40:04.777668 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:04.777597 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b77193b8-0b50-4d53-beb9-619be2ea30b5" containerName="istio-proxy" Oct 30 22:40:04.777668 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:04.777622 6006 cpu_manager.go:414] "containerMap: removing container" podUID="b77193b8-0b50-4d53-beb9-619be2ea30b5" containerName="istio-proxy" Oct 30 22:40:04.777668 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:04.777627 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b77193b8-0b50-4d53-beb9-619be2ea30b5" containerName="istio-proxy" Oct 30 22:40:04.777808 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:04.777709 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="b77193b8-0b50-4d53-beb9-619be2ea30b5" containerName="istio-proxy" Oct 30 22:40:04.777808 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:04.777721 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="21b585aa-5c59-491c-bc39-10f6afee58a3" containerName="revoke-keys" Oct 30 22:40:04.782456 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:04.782432 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg" Oct 30 22:40:04.792584 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:04.792550 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89506bb4-c9d4-43dc-9842-59f4295359ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg\" (UID: \"89506bb4-c9d4-43dc-9842-59f4295359ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg" Oct 30 22:40:04.792754 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:04.792603 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89506bb4-c9d4-43dc-9842-59f4295359ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg\" (UID: \"89506bb4-c9d4-43dc-9842-59f4295359ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg" Oct 30 22:40:04.792754 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:04.792716 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97r5z\" (UniqueName: \"kubernetes.io/projected/89506bb4-c9d4-43dc-9842-59f4295359ba-kube-api-access-97r5z\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg\" (UID: \"89506bb4-c9d4-43dc-9842-59f4295359ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg" Oct 30 22:40:04.836943 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:04.836905 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg"] Oct 30 22:40:04.880935 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:04.880902 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:40:04.881140 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:04.880955 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:40:04.881140 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:04.880972 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:40:04.881140 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:04.880991 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:40:04.893447 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:04.893405 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89506bb4-c9d4-43dc-9842-59f4295359ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg\" (UID: \"89506bb4-c9d4-43dc-9842-59f4295359ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg" Oct 30 22:40:04.893624 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:04.893465 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89506bb4-c9d4-43dc-9842-59f4295359ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg\" (UID: \"89506bb4-c9d4-43dc-9842-59f4295359ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg" Oct 30 22:40:04.893624 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:04.893505 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97r5z\" (UniqueName: \"kubernetes.io/projected/89506bb4-c9d4-43dc-9842-59f4295359ba-kube-api-access-97r5z\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg\" (UID: \"89506bb4-c9d4-43dc-9842-59f4295359ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg" Oct 30 22:40:04.894096 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:04.894072 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89506bb4-c9d4-43dc-9842-59f4295359ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg\" (UID: \"89506bb4-c9d4-43dc-9842-59f4295359ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg" Oct 30 22:40:04.895936 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:04.895916 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89506bb4-c9d4-43dc-9842-59f4295359ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg\" (UID: \"89506bb4-c9d4-43dc-9842-59f4295359ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg" Oct 30 22:40:04.952015 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:04.951983 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97r5z\" (UniqueName: \"kubernetes.io/projected/89506bb4-c9d4-43dc-9842-59f4295359ba-kube-api-access-97r5z\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg\" (UID: \"89506bb4-c9d4-43dc-9842-59f4295359ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg" Oct 30 22:40:05.095374 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:05.095335 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg" Oct 30 22:40:05.471337 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:05.471302 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg"] Oct 30 22:40:05.474668 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:40:05.474634 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89506bb4_c9d4_43dc_9842_59f4295359ba.slice/crio-685a24e7e212970696d88ef6deda898513c53644010680ebe97853115a3d33cb WatchSource:0}: Error finding container 685a24e7e212970696d88ef6deda898513c53644010680ebe97853115a3d33cb: Status 404 returned error can't find the container with id 685a24e7e212970696d88ef6deda898513c53644010680ebe97853115a3d33cb Oct 30 22:40:05.998752 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:05.998710 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg" event={"ID":"89506bb4-c9d4-43dc-9842-59f4295359ba","Type":"ContainerStarted","Data":"d304753395bd9cb2cf5db23a1dbfc681f0f6798e9674665a9f1fc4e6f332295c"} Oct 30 22:40:05.999177 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:05.998780 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg" event={"ID":"89506bb4-c9d4-43dc-9842-59f4295359ba","Type":"ContainerStarted","Data":"685a24e7e212970696d88ef6deda898513c53644010680ebe97853115a3d33cb"} Oct 30 22:40:06.119361 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:06.119232 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg" podStartSLOduration=2.119219211 podStartE2EDuration="2.119219211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:06.117101342 +0000 UTC m=+3008.669277654" watchObservedRunningTime="2026-10-30 22:40:06.119219211 +0000 UTC m=+3008.671395511" Oct 30 22:40:06.506117 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:06.506064 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:40:07.006263 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:07.006212 6006 generic.go:356] "Generic (PLEG): container finished" podID="89506bb4-c9d4-43dc-9842-59f4295359ba" containerID="d304753395bd9cb2cf5db23a1dbfc681f0f6798e9674665a9f1fc4e6f332295c" exitCode=0 Oct 30 22:40:07.006684 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:07.006306 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg" event={"ID":"89506bb4-c9d4-43dc-9842-59f4295359ba","Type":"ContainerDied","Data":"d304753395bd9cb2cf5db23a1dbfc681f0f6798e9674665a9f1fc4e6f332295c"} Oct 30 22:40:08.169690 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:08.169660 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg" Oct 30 22:40:08.233031 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:08.232970 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89506bb4-c9d4-43dc-9842-59f4295359ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89506bb4-c9d4-43dc-9842-59f4295359ba-maas-api-openshift-ca\") pod \"89506bb4-c9d4-43dc-9842-59f4295359ba\" (UID: \"89506bb4-c9d4-43dc-9842-59f4295359ba\") " Oct 30 22:40:08.233277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:08.233068 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89506bb4-c9d4-43dc-9842-59f4295359ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89506bb4-c9d4-43dc-9842-59f4295359ba-maas-api-xks-ca\") pod \"89506bb4-c9d4-43dc-9842-59f4295359ba\" (UID: \"89506bb4-c9d4-43dc-9842-59f4295359ba\") " Oct 30 22:40:08.233277 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:08.233131 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89506bb4-c9d4-43dc-9842-59f4295359ba-kube-api-access-97r5z\" (UniqueName: \"kubernetes.io/projected/89506bb4-c9d4-43dc-9842-59f4295359ba-kube-api-access-97r5z\") pod \"89506bb4-c9d4-43dc-9842-59f4295359ba\" (UID: \"89506bb4-c9d4-43dc-9842-59f4295359ba\") " Oct 30 22:40:08.233478 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:08.233452 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89506bb4-c9d4-43dc-9842-59f4295359ba-maas-api-openshift-ca" pod "89506bb4-c9d4-43dc-9842-59f4295359ba" (UID: "89506bb4-c9d4-43dc-9842-59f4295359ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:08.235510 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:08.235481 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89506bb4-c9d4-43dc-9842-59f4295359ba-maas-api-xks-ca" pod "89506bb4-c9d4-43dc-9842-59f4295359ba" (UID: "89506bb4-c9d4-43dc-9842-59f4295359ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:40:08.235656 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:08.235583 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89506bb4-c9d4-43dc-9842-59f4295359ba-kube-api-access-97r5z" pod "89506bb4-c9d4-43dc-9842-59f4295359ba" (UID: "89506bb4-c9d4-43dc-9842-59f4295359ba"). InnerVolumeSpecName "kube-api-access-97r5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:08.334604 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:08.334556 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89506bb4-c9d4-43dc-9842-59f4295359ba-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:08.334604 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:08.334595 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97r5z\" (UniqueName: \"kubernetes.io/projected/89506bb4-c9d4-43dc-9842-59f4295359ba-kube-api-access-97r5z\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:08.334604 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:08.334606 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89506bb4-c9d4-43dc-9842-59f4295359ba-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:09.023744 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:09.023701 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg" event={"ID":"89506bb4-c9d4-43dc-9842-59f4295359ba","Type":"ContainerDied","Data":"685a24e7e212970696d88ef6deda898513c53644010680ebe97853115a3d33cb"} Oct 30 22:40:09.023964 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:09.023765 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="685a24e7e212970696d88ef6deda898513c53644010680ebe97853115a3d33cb" Oct 30 22:40:09.023964 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:09.023714 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg" Oct 30 22:40:10.685376 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:10.685331 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc"] Oct 30 22:40:10.685911 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:10.685860 6006 cpu_manager.go:414] "containerMap: removing container" podUID="89506bb4-c9d4-43dc-9842-59f4295359ba" containerName="revoke-keys" Oct 30 22:40:10.685911 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:10.685880 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89506bb4-c9d4-43dc-9842-59f4295359ba" containerName="revoke-keys" Oct 30 22:40:10.686026 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:10.686012 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="b77193b8-0b50-4d53-beb9-619be2ea30b5" containerName="istio-proxy" Oct 30 22:40:10.686090 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:10.686024 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="89506bb4-c9d4-43dc-9842-59f4295359ba" containerName="revoke-keys" Oct 30 22:40:10.689683 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:10.689656 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc" Oct 30 22:40:10.741032 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:10.740998 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:40:10.741032 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:10.741003 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:40:10.741297 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:10.741146 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:40:10.759015 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:10.758972 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfp9v\" (UniqueName: \"kubernetes.io/projected/5248e340-c665-485b-b2a6-0a07d14d5f53-kube-api-access-gfp9v\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc\" (UID: \"5248e340-c665-485b-b2a6-0a07d14d5f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc" Oct 30 22:40:10.759207 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:10.759164 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5248e340-c665-485b-b2a6-0a07d14d5f53-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc\" (UID: \"5248e340-c665-485b-b2a6-0a07d14d5f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc" Oct 30 22:40:10.759304 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:10.759211 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5248e340-c665-485b-b2a6-0a07d14d5f53-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc\" (UID: \"5248e340-c665-485b-b2a6-0a07d14d5f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc" Oct 30 22:40:10.787767 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:10.787723 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:40:10.789640 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:10.789612 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc"] Oct 30 22:40:10.860530 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:10.860487 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfp9v\" (UniqueName: \"kubernetes.io/projected/5248e340-c665-485b-b2a6-0a07d14d5f53-kube-api-access-gfp9v\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc\" (UID: \"5248e340-c665-485b-b2a6-0a07d14d5f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc" Oct 30 22:40:10.860749 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:10.860588 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5248e340-c665-485b-b2a6-0a07d14d5f53-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc\" (UID: \"5248e340-c665-485b-b2a6-0a07d14d5f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc" Oct 30 22:40:10.860749 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:10.860625 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5248e340-c665-485b-b2a6-0a07d14d5f53-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc\" (UID: \"5248e340-c665-485b-b2a6-0a07d14d5f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc" Oct 30 22:40:10.861367 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:10.861337 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5248e340-c665-485b-b2a6-0a07d14d5f53-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc\" (UID: \"5248e340-c665-485b-b2a6-0a07d14d5f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc" Oct 30 22:40:10.863180 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:10.863157 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5248e340-c665-485b-b2a6-0a07d14d5f53-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc\" (UID: \"5248e340-c665-485b-b2a6-0a07d14d5f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc" Oct 30 22:40:10.919012 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:10.918977 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfp9v\" (UniqueName: \"kubernetes.io/projected/5248e340-c665-485b-b2a6-0a07d14d5f53-kube-api-access-gfp9v\") pod \"maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc\" (UID: \"5248e340-c665-485b-b2a6-0a07d14d5f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc" Oct 30 22:40:11.003492 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:11.003403 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc" Oct 30 22:40:11.377504 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:11.377477 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc"] Oct 30 22:40:11.379710 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:40:11.379679 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5248e340_c665_485b_b2a6_0a07d14d5f53.slice/crio-6ca2f0bb6054bcab79bed1a9d04ef85954625170065228f43e3e7628fff3d9d2 WatchSource:0}: Error finding container 6ca2f0bb6054bcab79bed1a9d04ef85954625170065228f43e3e7628fff3d9d2: Status 404 returned error can't find the container with id 6ca2f0bb6054bcab79bed1a9d04ef85954625170065228f43e3e7628fff3d9d2 Oct 30 22:40:12.045907 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:12.045865 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc" event={"ID":"5248e340-c665-485b-b2a6-0a07d14d5f53","Type":"ContainerStarted","Data":"f59d90a61649c0e533739321b8dc6e3461dea4043041fce4bf6c226b157ddc34"} Oct 30 22:40:12.046351 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:12.045933 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc" event={"ID":"5248e340-c665-485b-b2a6-0a07d14d5f53","Type":"ContainerStarted","Data":"6ca2f0bb6054bcab79bed1a9d04ef85954625170065228f43e3e7628fff3d9d2"} Oct 30 22:40:12.160938 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:12.160873 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc" podStartSLOduration=2.16085373 podStartE2EDuration="2.16085373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:12.159078042 +0000 UTC m=+3014.711254342" watchObservedRunningTime="2026-10-30 22:40:12.16085373 +0000 UTC m=+3014.713030031" Oct 30 22:40:13.055109 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:13.055071 6006 generic.go:356] "Generic (PLEG): container finished" podID="5248e340-c665-485b-b2a6-0a07d14d5f53" containerID="f59d90a61649c0e533739321b8dc6e3461dea4043041fce4bf6c226b157ddc34" exitCode=0 Oct 30 22:40:13.055577 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:13.055165 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc" event={"ID":"5248e340-c665-485b-b2a6-0a07d14d5f53","Type":"ContainerDied","Data":"f59d90a61649c0e533739321b8dc6e3461dea4043041fce4bf6c226b157ddc34"} Oct 30 22:40:14.207593 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:14.207560 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc" Oct 30 22:40:14.296677 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:14.296631 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5248e340-c665-485b-b2a6-0a07d14d5f53-kube-api-access-gfp9v\" (UniqueName: \"kubernetes.io/projected/5248e340-c665-485b-b2a6-0a07d14d5f53-kube-api-access-gfp9v\") pod \"5248e340-c665-485b-b2a6-0a07d14d5f53\" (UID: \"5248e340-c665-485b-b2a6-0a07d14d5f53\") " Oct 30 22:40:14.296883 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:14.296689 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5248e340-c665-485b-b2a6-0a07d14d5f53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5248e340-c665-485b-b2a6-0a07d14d5f53-maas-api-openshift-ca\") pod \"5248e340-c665-485b-b2a6-0a07d14d5f53\" (UID: \"5248e340-c665-485b-b2a6-0a07d14d5f53\") " Oct 30 22:40:14.296883 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:14.296797 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5248e340-c665-485b-b2a6-0a07d14d5f53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5248e340-c665-485b-b2a6-0a07d14d5f53-maas-api-xks-ca\") pod \"5248e340-c665-485b-b2a6-0a07d14d5f53\" (UID: \"5248e340-c665-485b-b2a6-0a07d14d5f53\") " Oct 30 22:40:14.297144 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:14.297114 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5248e340-c665-485b-b2a6-0a07d14d5f53-maas-api-openshift-ca" pod "5248e340-c665-485b-b2a6-0a07d14d5f53" (UID: "5248e340-c665-485b-b2a6-0a07d14d5f53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:14.298881 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:14.298852 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5248e340-c665-485b-b2a6-0a07d14d5f53-kube-api-access-gfp9v" pod "5248e340-c665-485b-b2a6-0a07d14d5f53" (UID: "5248e340-c665-485b-b2a6-0a07d14d5f53"). InnerVolumeSpecName "kube-api-access-gfp9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:14.298881 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:14.298853 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5248e340-c665-485b-b2a6-0a07d14d5f53-maas-api-xks-ca" pod "5248e340-c665-485b-b2a6-0a07d14d5f53" (UID: "5248e340-c665-485b-b2a6-0a07d14d5f53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:40:14.398638 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:14.398540 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5248e340-c665-485b-b2a6-0a07d14d5f53-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:14.398638 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:14.398577 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfp9v\" (UniqueName: \"kubernetes.io/projected/5248e340-c665-485b-b2a6-0a07d14d5f53-kube-api-access-gfp9v\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:14.398638 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:14.398590 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5248e340-c665-485b-b2a6-0a07d14d5f53-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:15.068659 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:15.068623 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc" Oct 30 22:40:15.068659 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:15.068638 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc" event={"ID":"5248e340-c665-485b-b2a6-0a07d14d5f53","Type":"ContainerDied","Data":"6ca2f0bb6054bcab79bed1a9d04ef85954625170065228f43e3e7628fff3d9d2"} Oct 30 22:40:15.068900 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:15.068683 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ca2f0bb6054bcab79bed1a9d04ef85954625170065228f43e3e7628fff3d9d2" Oct 30 22:40:20.229609 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:20.229569 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg"] Oct 30 22:40:20.275331 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:20.275277 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-943d8f-nfzkg"] Oct 30 22:40:21.038349 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:21.038304 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw"] Oct 30 22:40:21.604372 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:21.604330 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89506bb4-c9d4-43dc-9842-59f4295359ba" path="/var/lib/kubelet/pods/89506bb4-c9d4-43dc-9842-59f4295359ba/volumes" Oct 30 22:40:25.977935 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:25.977894 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:40:26.038764 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:26.038714 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw" podUID="a35ddaa8-6863-4c1f-b9e8-711a8e76e006" containerName="payload-processing" containerID="cri-o://cfcb71a25305fcc666a6a8f5d539f64d19b184b4678358ff364655ca1e23cbb5" gracePeriod=25 Oct 30 22:40:27.911221 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:40:27.911182 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:40:31.732102 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:31.732059 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw" podUID="a35ddaa8-6863-4c1f-b9e8-711a8e76e006" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:40:31.942283 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:31.940715 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10e61ea0-22k8q"] Oct 30 22:40:31.942283 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:31.941844 6006 cpu_manager.go:414] "containerMap: removing container" podUID="5248e340-c665-485b-b2a6-0a07d14d5f53" containerName="revoke-keys" Oct 30 22:40:31.942283 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:31.941868 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5248e340-c665-485b-b2a6-0a07d14d5f53" containerName="revoke-keys" Oct 30 22:40:31.942283 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:31.942155 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="5248e340-c665-485b-b2a6-0a07d14d5f53" containerName="revoke-keys" Oct 30 22:40:31.952291 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:31.952229 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e61ea0-22k8q" Oct 30 22:40:31.998512 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:31.998404 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10e61ea0-22k8q"] Oct 30 22:40:32.043294 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:32.043233 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:40:32.043294 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:32.043233 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:40:32.043545 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:32.043359 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:40:32.048013 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:32.047978 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:40:32.081651 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:32.081614 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1283cb-bd03-435b-b957-0721eb8995c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10e61ea0-22k8q\" (UID: \"9b1283cb-bd03-435b-b957-0721eb8995c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e61ea0-22k8q" Oct 30 22:40:32.081651 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:32.081655 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1283cb-bd03-435b-b957-0721eb8995c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10e61ea0-22k8q\" (UID: \"9b1283cb-bd03-435b-b957-0721eb8995c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e61ea0-22k8q" Oct 30 22:40:32.081888 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:32.081758 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7rbj\" (UniqueName: \"kubernetes.io/projected/9b1283cb-bd03-435b-b957-0721eb8995c4-kube-api-access-m7rbj\") pod \"maas-api-revoke-sub-10e61ea0-22k8q\" (UID: \"9b1283cb-bd03-435b-b957-0721eb8995c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e61ea0-22k8q" Oct 30 22:40:32.182410 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:32.182367 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1283cb-bd03-435b-b957-0721eb8995c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10e61ea0-22k8q\" (UID: \"9b1283cb-bd03-435b-b957-0721eb8995c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e61ea0-22k8q" Oct 30 22:40:32.182645 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:32.182418 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1283cb-bd03-435b-b957-0721eb8995c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10e61ea0-22k8q\" (UID: \"9b1283cb-bd03-435b-b957-0721eb8995c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e61ea0-22k8q" Oct 30 22:40:32.182645 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:32.182584 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7rbj\" (UniqueName: \"kubernetes.io/projected/9b1283cb-bd03-435b-b957-0721eb8995c4-kube-api-access-m7rbj\") pod \"maas-api-revoke-sub-10e61ea0-22k8q\" (UID: \"9b1283cb-bd03-435b-b957-0721eb8995c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e61ea0-22k8q" Oct 30 22:40:32.183183 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:32.183156 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1283cb-bd03-435b-b957-0721eb8995c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10e61ea0-22k8q\" (UID: \"9b1283cb-bd03-435b-b957-0721eb8995c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e61ea0-22k8q" Oct 30 22:40:32.184839 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:32.184820 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1283cb-bd03-435b-b957-0721eb8995c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10e61ea0-22k8q\" (UID: \"9b1283cb-bd03-435b-b957-0721eb8995c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e61ea0-22k8q" Oct 30 22:40:32.241388 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:32.241341 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7rbj\" (UniqueName: \"kubernetes.io/projected/9b1283cb-bd03-435b-b957-0721eb8995c4-kube-api-access-m7rbj\") pod \"maas-api-revoke-sub-10e61ea0-22k8q\" (UID: \"9b1283cb-bd03-435b-b957-0721eb8995c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e61ea0-22k8q" Oct 30 22:40:32.266831 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:32.266745 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e61ea0-22k8q" Oct 30 22:40:32.646729 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:32.646704 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10e61ea0-22k8q"] Oct 30 22:40:32.650205 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:40:32.650172 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b1283cb_bd03_435b_b957_0721eb8995c4.slice/crio-eac47c617f43b33741278a65c795fc1dad4e99aef83eed50fab86b7a176557c3 WatchSource:0}: Error finding container eac47c617f43b33741278a65c795fc1dad4e99aef83eed50fab86b7a176557c3: Status 404 returned error can't find the container with id eac47c617f43b33741278a65c795fc1dad4e99aef83eed50fab86b7a176557c3 Oct 30 22:40:32.672227 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:32.672202 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886"] Oct 30 22:40:32.677110 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:32.677082 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn"] Oct 30 22:40:32.688566 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:32.688499 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn"] Oct 30 22:40:33.183772 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:33.183684 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e61ea0-22k8q" event={"ID":"9b1283cb-bd03-435b-b957-0721eb8995c4","Type":"ContainerStarted","Data":"771b30343eb77bf997e51a3a212d2c96b54104b4334fa886480a3fc50f9a7fa5"} Oct 30 22:40:33.183772 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:33.183737 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e61ea0-22k8q" event={"ID":"9b1283cb-bd03-435b-b957-0721eb8995c4","Type":"ContainerStarted","Data":"eac47c617f43b33741278a65c795fc1dad4e99aef83eed50fab86b7a176557c3"} Oct 30 22:40:33.293646 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:33.293584 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e61ea0-22k8q" podStartSLOduration=2.2935642769999998 podStartE2EDuration="2.293564277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:33.292886246 +0000 UTC m=+3035.845062548" watchObservedRunningTime="2026-10-30 22:40:33.293564277 +0000 UTC m=+3035.845740579" Oct 30 22:40:34.191163 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:34.191126 6006 generic.go:356] "Generic (PLEG): container finished" podID="9b1283cb-bd03-435b-b957-0721eb8995c4" containerID="771b30343eb77bf997e51a3a212d2c96b54104b4334fa886480a3fc50f9a7fa5" exitCode=0 Oct 30 22:40:34.191633 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:34.191210 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e61ea0-22k8q" event={"ID":"9b1283cb-bd03-435b-b957-0721eb8995c4","Type":"ContainerDied","Data":"771b30343eb77bf997e51a3a212d2c96b54104b4334fa886480a3fc50f9a7fa5"} Oct 30 22:40:35.353651 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:35.353623 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e61ea0-22k8q" Oct 30 22:40:35.518273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:35.518155 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b1283cb-bd03-435b-b957-0721eb8995c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1283cb-bd03-435b-b957-0721eb8995c4-maas-api-openshift-ca\") pod \"9b1283cb-bd03-435b-b957-0721eb8995c4\" (UID: \"9b1283cb-bd03-435b-b957-0721eb8995c4\") " Oct 30 22:40:35.518273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:35.518206 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b1283cb-bd03-435b-b957-0721eb8995c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1283cb-bd03-435b-b957-0721eb8995c4-maas-api-xks-ca\") pod \"9b1283cb-bd03-435b-b957-0721eb8995c4\" (UID: \"9b1283cb-bd03-435b-b957-0721eb8995c4\") " Oct 30 22:40:35.518469 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:35.518391 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b1283cb-bd03-435b-b957-0721eb8995c4-kube-api-access-m7rbj\" (UniqueName: \"kubernetes.io/projected/9b1283cb-bd03-435b-b957-0721eb8995c4-kube-api-access-m7rbj\") pod \"9b1283cb-bd03-435b-b957-0721eb8995c4\" (UID: \"9b1283cb-bd03-435b-b957-0721eb8995c4\") " Oct 30 22:40:35.518680 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:35.518654 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b1283cb-bd03-435b-b957-0721eb8995c4-maas-api-openshift-ca" pod "9b1283cb-bd03-435b-b957-0721eb8995c4" (UID: "9b1283cb-bd03-435b-b957-0721eb8995c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:35.520552 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:35.520516 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b1283cb-bd03-435b-b957-0721eb8995c4-maas-api-xks-ca" pod "9b1283cb-bd03-435b-b957-0721eb8995c4" (UID: "9b1283cb-bd03-435b-b957-0721eb8995c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:40:35.520731 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:35.520708 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b1283cb-bd03-435b-b957-0721eb8995c4-kube-api-access-m7rbj" pod "9b1283cb-bd03-435b-b957-0721eb8995c4" (UID: "9b1283cb-bd03-435b-b957-0721eb8995c4"). InnerVolumeSpecName "kube-api-access-m7rbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:35.619975 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:35.619944 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1283cb-bd03-435b-b957-0721eb8995c4-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:35.620147 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:35.619978 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1283cb-bd03-435b-b957-0721eb8995c4-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:35.620147 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:35.619993 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7rbj\" (UniqueName: \"kubernetes.io/projected/9b1283cb-bd03-435b-b957-0721eb8995c4-kube-api-access-m7rbj\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:36.204332 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:36.204299 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e61ea0-22k8q" event={"ID":"9b1283cb-bd03-435b-b957-0721eb8995c4","Type":"ContainerDied","Data":"eac47c617f43b33741278a65c795fc1dad4e99aef83eed50fab86b7a176557c3"} Oct 30 22:40:36.204518 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:36.204351 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eac47c617f43b33741278a65c795fc1dad4e99aef83eed50fab86b7a176557c3" Oct 30 22:40:36.204518 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:36.204307 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e61ea0-22k8q" Oct 30 22:40:38.628613 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:38.628571 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99"] Oct 30 22:40:41.732961 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:41.732921 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw" podUID="a35ddaa8-6863-4c1f-b9e8-711a8e76e006" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:40:41.754674 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:41.754633 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-869d3806-cvr54"] Oct 30 22:40:41.755177 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:41.755156 6006 cpu_manager.go:414] "containerMap: removing container" podUID="9b1283cb-bd03-435b-b957-0721eb8995c4" containerName="revoke-keys" Oct 30 22:40:41.755311 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:41.755179 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b1283cb-bd03-435b-b957-0721eb8995c4" containerName="revoke-keys" Oct 30 22:40:41.755399 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:41.755382 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b1283cb-bd03-435b-b957-0721eb8995c4" containerName="revoke-keys" Oct 30 22:40:41.759121 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:41.759092 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d3806-cvr54" Oct 30 22:40:41.814301 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:41.814260 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-869d3806-cvr54"] Oct 30 22:40:41.859053 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:41.859020 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:40:41.859308 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:41.859080 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:40:41.859308 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:41.859201 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:40:41.859426 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:41.859348 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:40:41.885824 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:41.885775 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h42x4\" (UniqueName: \"kubernetes.io/projected/7891883f-2be5-4384-bdd3-861df80fd8fc-kube-api-access-h42x4\") pod \"maas-api-revoke-sub-869d3806-cvr54\" (UID: \"7891883f-2be5-4384-bdd3-861df80fd8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d3806-cvr54" Oct 30 22:40:41.886002 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:41.885891 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7891883f-2be5-4384-bdd3-861df80fd8fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-869d3806-cvr54\" (UID: \"7891883f-2be5-4384-bdd3-861df80fd8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d3806-cvr54" Oct 30 22:40:41.886002 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:41.885991 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7891883f-2be5-4384-bdd3-861df80fd8fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-869d3806-cvr54\" (UID: \"7891883f-2be5-4384-bdd3-861df80fd8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d3806-cvr54" Oct 30 22:40:41.987224 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:41.987116 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7891883f-2be5-4384-bdd3-861df80fd8fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-869d3806-cvr54\" (UID: \"7891883f-2be5-4384-bdd3-861df80fd8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d3806-cvr54" Oct 30 22:40:41.987463 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:41.987282 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7891883f-2be5-4384-bdd3-861df80fd8fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-869d3806-cvr54\" (UID: \"7891883f-2be5-4384-bdd3-861df80fd8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d3806-cvr54" Oct 30 22:40:41.987463 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:41.987375 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h42x4\" (UniqueName: \"kubernetes.io/projected/7891883f-2be5-4384-bdd3-861df80fd8fc-kube-api-access-h42x4\") pod \"maas-api-revoke-sub-869d3806-cvr54\" (UID: \"7891883f-2be5-4384-bdd3-861df80fd8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d3806-cvr54" Oct 30 22:40:41.988347 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:41.988321 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7891883f-2be5-4384-bdd3-861df80fd8fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-869d3806-cvr54\" (UID: \"7891883f-2be5-4384-bdd3-861df80fd8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d3806-cvr54" Oct 30 22:40:41.990204 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:41.990174 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7891883f-2be5-4384-bdd3-861df80fd8fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-869d3806-cvr54\" (UID: \"7891883f-2be5-4384-bdd3-861df80fd8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d3806-cvr54" Oct 30 22:40:42.050914 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:42.050879 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h42x4\" (UniqueName: \"kubernetes.io/projected/7891883f-2be5-4384-bdd3-861df80fd8fc-kube-api-access-h42x4\") pod \"maas-api-revoke-sub-869d3806-cvr54\" (UID: \"7891883f-2be5-4384-bdd3-861df80fd8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d3806-cvr54" Oct 30 22:40:42.075051 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:42.075019 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d3806-cvr54" Oct 30 22:40:42.453047 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:42.453015 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-869d3806-cvr54"] Oct 30 22:40:42.456068 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:40:42.456038 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7891883f_2be5_4384_bdd3_861df80fd8fc.slice/crio-5a2100d3b2d56f8d0148e577c75d839a3ebd7c820fcacdd0ca64f97d17059995 WatchSource:0}: Error finding container 5a2100d3b2d56f8d0148e577c75d839a3ebd7c820fcacdd0ca64f97d17059995: Status 404 returned error can't find the container with id 5a2100d3b2d56f8d0148e577c75d839a3ebd7c820fcacdd0ca64f97d17059995 Oct 30 22:40:43.252647 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:43.252602 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d3806-cvr54" event={"ID":"7891883f-2be5-4384-bdd3-861df80fd8fc","Type":"ContainerStarted","Data":"5b4839d67c0ecb9a8a3621ef7f4a7496dce5e956facffcda44a1edb821de5a6b"} Oct 30 22:40:43.253073 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:43.252691 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d3806-cvr54" event={"ID":"7891883f-2be5-4384-bdd3-861df80fd8fc","Type":"ContainerStarted","Data":"5a2100d3b2d56f8d0148e577c75d839a3ebd7c820fcacdd0ca64f97d17059995"} Oct 30 22:40:43.363128 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:43.362893 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d3806-cvr54" podStartSLOduration=2.362874002 podStartE2EDuration="2.362874002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:43.362044967 +0000 UTC m=+3045.914221267" watchObservedRunningTime="2026-10-30 22:40:43.362874002 +0000 UTC m=+3045.915050301" Oct 30 22:40:44.225637 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:44.225590 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99" podUID="f9a2a7a5-468b-43f3-8528-1cc78b307324" containerName="payload-processing" containerID="cri-o://ea914c779525256ba30f6a32d4a6345f873203fb025f8c1e655dcc2e36fd40e4" gracePeriod=25 Oct 30 22:40:44.260410 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:44.260368 6006 generic.go:356] "Generic (PLEG): container finished" podID="7891883f-2be5-4384-bdd3-861df80fd8fc" containerID="5b4839d67c0ecb9a8a3621ef7f4a7496dce5e956facffcda44a1edb821de5a6b" exitCode=0 Oct 30 22:40:44.260850 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:44.260466 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d3806-cvr54" event={"ID":"7891883f-2be5-4384-bdd3-861df80fd8fc","Type":"ContainerDied","Data":"5b4839d67c0ecb9a8a3621ef7f4a7496dce5e956facffcda44a1edb821de5a6b"} Oct 30 22:40:45.416765 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:45.416725 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d3806-cvr54" Oct 30 22:40:45.421692 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:45.421661 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7891883f-2be5-4384-bdd3-861df80fd8fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7891883f-2be5-4384-bdd3-861df80fd8fc-maas-api-openshift-ca\") pod \"7891883f-2be5-4384-bdd3-861df80fd8fc\" (UID: \"7891883f-2be5-4384-bdd3-861df80fd8fc\") " Oct 30 22:40:45.421850 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:45.421712 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7891883f-2be5-4384-bdd3-861df80fd8fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7891883f-2be5-4384-bdd3-861df80fd8fc-maas-api-xks-ca\") pod \"7891883f-2be5-4384-bdd3-861df80fd8fc\" (UID: \"7891883f-2be5-4384-bdd3-861df80fd8fc\") " Oct 30 22:40:45.421850 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:45.421784 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7891883f-2be5-4384-bdd3-861df80fd8fc-kube-api-access-h42x4\" (UniqueName: \"kubernetes.io/projected/7891883f-2be5-4384-bdd3-861df80fd8fc-kube-api-access-h42x4\") pod \"7891883f-2be5-4384-bdd3-861df80fd8fc\" (UID: \"7891883f-2be5-4384-bdd3-861df80fd8fc\") " Oct 30 22:40:45.422060 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:45.422017 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7891883f-2be5-4384-bdd3-861df80fd8fc-maas-api-openshift-ca" pod "7891883f-2be5-4384-bdd3-861df80fd8fc" (UID: "7891883f-2be5-4384-bdd3-861df80fd8fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:45.423816 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:45.423784 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7891883f-2be5-4384-bdd3-861df80fd8fc-maas-api-xks-ca" pod "7891883f-2be5-4384-bdd3-861df80fd8fc" (UID: "7891883f-2be5-4384-bdd3-861df80fd8fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:40:45.423942 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:45.423860 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7891883f-2be5-4384-bdd3-861df80fd8fc-kube-api-access-h42x4" pod "7891883f-2be5-4384-bdd3-861df80fd8fc" (UID: "7891883f-2be5-4384-bdd3-861df80fd8fc"). InnerVolumeSpecName "kube-api-access-h42x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:45.522572 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:45.522467 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h42x4\" (UniqueName: \"kubernetes.io/projected/7891883f-2be5-4384-bdd3-861df80fd8fc-kube-api-access-h42x4\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:45.522572 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:45.522503 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7891883f-2be5-4384-bdd3-861df80fd8fc-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:45.522572 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:45.522514 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7891883f-2be5-4384-bdd3-861df80fd8fc-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:46.276576 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:46.276487 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d3806-cvr54" Oct 30 22:40:46.276576 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:46.276509 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d3806-cvr54" event={"ID":"7891883f-2be5-4384-bdd3-861df80fd8fc","Type":"ContainerDied","Data":"5a2100d3b2d56f8d0148e577c75d839a3ebd7c820fcacdd0ca64f97d17059995"} Oct 30 22:40:46.276576 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:46.276570 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a2100d3b2d56f8d0148e577c75d839a3ebd7c820fcacdd0ca64f97d17059995" Oct 30 22:40:46.278590 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:46.278558 6006 generic.go:356] "Generic (PLEG): container finished" podID="a35ddaa8-6863-4c1f-b9e8-711a8e76e006" containerID="cfcb71a25305fcc666a6a8f5d539f64d19b184b4678358ff364655ca1e23cbb5" exitCode=0 Oct 30 22:40:46.278786 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:46.278620 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw" event={"ID":"a35ddaa8-6863-4c1f-b9e8-711a8e76e006","Type":"ContainerDied","Data":"cfcb71a25305fcc666a6a8f5d539f64d19b184b4678358ff364655ca1e23cbb5"} Oct 30 22:40:46.294681 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:46.294613 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw" Oct 30 22:40:46.329982 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:46.329944 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a35ddaa8-6863-4c1f-b9e8-711a8e76e006-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a35ddaa8-6863-4c1f-b9e8-711a8e76e006-plugins-config-volume\") pod \"a35ddaa8-6863-4c1f-b9e8-711a8e76e006\" (UID: \"a35ddaa8-6863-4c1f-b9e8-711a8e76e006\") " Oct 30 22:40:46.330187 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:46.329988 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a35ddaa8-6863-4c1f-b9e8-711a8e76e006-kube-api-access-4hd84\" (UniqueName: \"kubernetes.io/projected/a35ddaa8-6863-4c1f-b9e8-711a8e76e006-kube-api-access-4hd84\") pod \"a35ddaa8-6863-4c1f-b9e8-711a8e76e006\" (UID: \"a35ddaa8-6863-4c1f-b9e8-711a8e76e006\") " Oct 30 22:40:46.330397 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:46.330375 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a35ddaa8-6863-4c1f-b9e8-711a8e76e006-plugins-config-volume" pod "a35ddaa8-6863-4c1f-b9e8-711a8e76e006" (UID: "a35ddaa8-6863-4c1f-b9e8-711a8e76e006"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:46.332222 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:46.332202 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a35ddaa8-6863-4c1f-b9e8-711a8e76e006-kube-api-access-4hd84" pod "a35ddaa8-6863-4c1f-b9e8-711a8e76e006" (UID: "a35ddaa8-6863-4c1f-b9e8-711a8e76e006"). InnerVolumeSpecName "kube-api-access-4hd84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:46.431727 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:46.431687 6006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a35ddaa8-6863-4c1f-b9e8-711a8e76e006-plugins-config-volume\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:46.431727 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:46.431721 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hd84\" (UniqueName: \"kubernetes.io/projected/a35ddaa8-6863-4c1f-b9e8-711a8e76e006-kube-api-access-4hd84\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:46.703874 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:46.703792 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99" podUID="f9a2a7a5-468b-43f3-8528-1cc78b307324" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:40:47.285943 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:47.285905 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw" event={"ID":"a35ddaa8-6863-4c1f-b9e8-711a8e76e006","Type":"ContainerDied","Data":"6b52d18cbd5930ca22fc4bbcfa426357ad19fec5574e688808b0cc3c2c9f528d"} Oct 30 22:40:47.286121 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:47.285968 6006 scope.go:122] "RemoveContainer" containerID="cfcb71a25305fcc666a6a8f5d539f64d19b184b4678358ff364655ca1e23cbb5" Oct 30 22:40:47.286121 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:47.285916 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw" Oct 30 22:40:47.441430 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:47.441395 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw"] Oct 30 22:40:47.489129 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:47.489089 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-fb0b93-76f5b9fcd7-twfdw"] Oct 30 22:40:47.602574 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:47.602536 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a35ddaa8-6863-4c1f-b9e8-711a8e76e006" path="/var/lib/kubelet/pods/a35ddaa8-6863-4c1f-b9e8-711a8e76e006/volumes" Oct 30 22:40:47.695875 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:47.695838 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" containerID="cri-o://da7156d8339ec9488e38599efafa17081e9925750125c778e6d950a7ded1e255" gracePeriod=45 Oct 30 22:40:47.697706 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:47.697644 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" containerID="cri-o://c0d8465ee82e660a07575a51d1134dccdc1b9a394f57c5ee90613a704e83d89e" gracePeriod=45 Oct 30 22:40:47.800411 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:47.800368 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" containerID="cri-o://3f9f2b4ae0b8cee22c4c7b599a6def74ca28b91c76c68bcb6e74a1107238c824" gracePeriod=45 Oct 30 22:40:47.817999 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:47.817965 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": dial tcp 10.132.0.66:8000: connect: connection refused" Oct 30 22:40:47.977289 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:47.977260 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:40:48.012111 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.012073 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:40:48.051370 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.051340 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-dshm\") pod \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\" (UID: \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\") " Oct 30 22:40:48.051553 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.051385 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-kube-api-access-8fjhm\" (UniqueName: \"kubernetes.io/projected/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-kube-api-access-8fjhm\") pod \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\" (UID: \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\") " Oct 30 22:40:48.051553 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.051434 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-kube-api-access-nphht\" (UniqueName: \"kubernetes.io/projected/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-kube-api-access-nphht\") pod \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\" (UID: \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\") " Oct 30 22:40:48.051553 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.051510 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-model-cache\") pod \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\" (UID: \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\") " Oct 30 22:40:48.051742 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.051558 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-dshm\") pod \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\" (UID: \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\") " Oct 30 22:40:48.051742 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.051602 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-tmp-dir\") pod \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\" (UID: \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\") " Oct 30 22:40:48.051742 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.051629 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-model-cache\") pod \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\" (UID: \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\") " Oct 30 22:40:48.051742 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.051670 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-tmp-dir\") pod \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\" (UID: \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\") " Oct 30 22:40:48.051742 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.051701 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-home\" (UniqueName: \"kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-home\") pod \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\" (UID: \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\") " Oct 30 22:40:48.052021 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.051743 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-tls-certs\" (UniqueName: \"kubernetes.io/secret/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-tls-certs\") pod \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\" (UID: \"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b\") " Oct 30 22:40:48.052021 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.051810 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-home\" (UniqueName: \"kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-home\") pod \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\" (UID: \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\") " Oct 30 22:40:48.052021 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.051871 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-tls-certs\") pod \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\" (UID: \"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a\") " Oct 30 22:40:48.052468 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.052440 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-tmp-dir" pod "d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" (UID: "d2715f84-b3fa-4d3d-80f9-18b5c8a7359a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:48.052673 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.052651 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-model-cache" pod "d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" (UID: "d2715f84-b3fa-4d3d-80f9-18b5c8a7359a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:48.054583 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.053187 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-tmp-dir" pod "b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" (UID: "b90b6498-0e99-4a90-b2ae-1bc4f2446d5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:48.054583 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.053407 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-model-cache" pod "b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" (UID: "b90b6498-0e99-4a90-b2ae-1bc4f2446d5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:48.054583 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.053644 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-home" pod "d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" (UID: "d2715f84-b3fa-4d3d-80f9-18b5c8a7359a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:48.054583 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.054016 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-home" pod "b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" (UID: "b90b6498-0e99-4a90-b2ae-1bc4f2446d5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:48.054583 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.054459 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-dshm" pod "b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" (UID: "b90b6498-0e99-4a90-b2ae-1bc4f2446d5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:48.055361 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.055327 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-dshm" pod "d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" (UID: "d2715f84-b3fa-4d3d-80f9-18b5c8a7359a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:48.055361 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.055337 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-kube-api-access-nphht" pod "b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" (UID: "b90b6498-0e99-4a90-b2ae-1bc4f2446d5b"). InnerVolumeSpecName "kube-api-access-nphht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:48.055764 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.055728 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-tls-certs" pod "d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" (UID: "d2715f84-b3fa-4d3d-80f9-18b5c8a7359a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:40:48.057070 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.056997 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-kube-api-access-8fjhm" pod "d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" (UID: "d2715f84-b3fa-4d3d-80f9-18b5c8a7359a"). InnerVolumeSpecName "kube-api-access-8fjhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:48.058187 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.058160 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:40:48.058647 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.058618 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-tls-certs" pod "b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" (UID: "b90b6498-0e99-4a90-b2ae-1bc4f2446d5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:40:48.153630 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.153551 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c71fda3-adf4-4b08-8a27-0065ac056a1d-kube-api-access-vmk98\" (UniqueName: \"kubernetes.io/projected/8c71fda3-adf4-4b08-8a27-0065ac056a1d-kube-api-access-vmk98\") pod \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\" (UID: \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\") " Oct 30 22:40:48.153630 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.153615 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-home\" (UniqueName: \"kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-home\") pod \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\" (UID: \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\") " Oct 30 22:40:48.153878 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.153664 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-dshm\") pod \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\" (UID: \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\") " Oct 30 22:40:48.153878 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.153686 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-model-cache\") pod \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\" (UID: \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\") " Oct 30 22:40:48.153878 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.153723 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-tmp-dir\") pod \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\" (UID: \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\") " Oct 30 22:40:48.153878 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.153753 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c71fda3-adf4-4b08-8a27-0065ac056a1d-tls-certs\" (UniqueName: \"kubernetes.io/secret/8c71fda3-adf4-4b08-8a27-0065ac056a1d-tls-certs\") pod \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\" (UID: \"8c71fda3-adf4-4b08-8a27-0065ac056a1d\") " Oct 30 22:40:48.154110 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.153945 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-home" pod "8c71fda3-adf4-4b08-8a27-0065ac056a1d" (UID: "8c71fda3-adf4-4b08-8a27-0065ac056a1d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:48.154110 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.153971 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-model-cache" pod "8c71fda3-adf4-4b08-8a27-0065ac056a1d" (UID: "8c71fda3-adf4-4b08-8a27-0065ac056a1d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:48.154110 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.154071 6006 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-tls-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:48.154263 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.154120 6006 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-home\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:48.154263 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.154134 6006 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-model-cache\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:48.154263 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.154149 6006 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-tls-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:48.154263 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.154162 6006 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-dshm\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:48.154263 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.154158 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-tmp-dir" pod "8c71fda3-adf4-4b08-8a27-0065ac056a1d" (UID: "8c71fda3-adf4-4b08-8a27-0065ac056a1d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:48.154263 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.154176 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fjhm\" (UniqueName: \"kubernetes.io/projected/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-kube-api-access-8fjhm\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:48.154263 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.154189 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nphht\" (UniqueName: \"kubernetes.io/projected/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-kube-api-access-nphht\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:48.154263 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.154200 6006 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-model-cache\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:48.154263 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.154212 6006 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-dshm\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:48.154263 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.154225 6006 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-tmp-dir\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:48.154263 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.154236 6006 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-model-cache\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:48.154263 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.154264 6006 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a-tmp-dir\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:48.154698 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.154277 6006 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b-home\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:48.154698 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.154289 6006 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-home\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:48.155901 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.155868 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c71fda3-adf4-4b08-8a27-0065ac056a1d-tls-certs" pod "8c71fda3-adf4-4b08-8a27-0065ac056a1d" (UID: "8c71fda3-adf4-4b08-8a27-0065ac056a1d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:40:48.156004 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.155906 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c71fda3-adf4-4b08-8a27-0065ac056a1d-kube-api-access-vmk98" pod "8c71fda3-adf4-4b08-8a27-0065ac056a1d" (UID: "8c71fda3-adf4-4b08-8a27-0065ac056a1d"). InnerVolumeSpecName "kube-api-access-vmk98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:48.156123 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.156107 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-dshm" pod "8c71fda3-adf4-4b08-8a27-0065ac056a1d" (UID: "8c71fda3-adf4-4b08-8a27-0065ac056a1d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:48.255222 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.255189 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmk98\" (UniqueName: \"kubernetes.io/projected/8c71fda3-adf4-4b08-8a27-0065ac056a1d-kube-api-access-vmk98\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:48.255222 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.255220 6006 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-dshm\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:48.255463 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.255234 6006 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c71fda3-adf4-4b08-8a27-0065ac056a1d-tmp-dir\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:48.255463 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.255245 6006 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c71fda3-adf4-4b08-8a27-0065ac056a1d-tls-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:48.294812 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.294772 6006 generic.go:356] "Generic (PLEG): container finished" podID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerID="da7156d8339ec9488e38599efafa17081e9925750125c778e6d950a7ded1e255" exitCode=0 Oct 30 22:40:48.295012 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.294824 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" Oct 30 22:40:48.295012 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.294865 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" event={"ID":"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a","Type":"ContainerDied","Data":"da7156d8339ec9488e38599efafa17081e9925750125c778e6d950a7ded1e255"} Oct 30 22:40:48.295012 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.294907 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886" event={"ID":"d2715f84-b3fa-4d3d-80f9-18b5c8a7359a","Type":"ContainerDied","Data":"91b6bb3f7ba7ba763fdbeb4f47bc8166ecb52a589cbe2f5623d32ad979f65233"} Oct 30 22:40:48.295012 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.294932 6006 scope.go:122] "RemoveContainer" containerID="da7156d8339ec9488e38599efafa17081e9925750125c778e6d950a7ded1e255" Oct 30 22:40:48.297357 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.297335 6006 generic.go:356] "Generic (PLEG): container finished" podID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerID="3f9f2b4ae0b8cee22c4c7b599a6def74ca28b91c76c68bcb6e74a1107238c824" exitCode=0 Oct 30 22:40:48.297476 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.297448 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" Oct 30 22:40:48.297476 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.297461 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" event={"ID":"8c71fda3-adf4-4b08-8a27-0065ac056a1d","Type":"ContainerDied","Data":"3f9f2b4ae0b8cee22c4c7b599a6def74ca28b91c76c68bcb6e74a1107238c824"} Oct 30 22:40:48.297595 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.297492 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" event={"ID":"8c71fda3-adf4-4b08-8a27-0065ac056a1d","Type":"ContainerDied","Data":"a3956a936c8b7b0603af8c109c7b8c3f086553b549dc6502eef3b77add552c06"} Oct 30 22:40:48.298992 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.298958 6006 generic.go:356] "Generic (PLEG): container finished" podID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerID="c0d8465ee82e660a07575a51d1134dccdc1b9a394f57c5ee90613a704e83d89e" exitCode=0 Oct 30 22:40:48.299095 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.299000 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" Oct 30 22:40:48.299095 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.299003 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" event={"ID":"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b","Type":"ContainerDied","Data":"c0d8465ee82e660a07575a51d1134dccdc1b9a394f57c5ee90613a704e83d89e"} Oct 30 22:40:48.299095 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.299031 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn" event={"ID":"b90b6498-0e99-4a90-b2ae-1bc4f2446d5b","Type":"ContainerDied","Data":"33049ee133310a074ae59cc2a097032e096608bb3e3d4e3b968871bf168e241c"} Oct 30 22:40:48.306552 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.306133 6006 scope.go:122] "RemoveContainer" containerID="da7156d8339ec9488e38599efafa17081e9925750125c778e6d950a7ded1e255" Oct 30 22:40:48.306552 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:40:48.306494 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da7156d8339ec9488e38599efafa17081e9925750125c778e6d950a7ded1e255\": container with ID starting with da7156d8339ec9488e38599efafa17081e9925750125c778e6d950a7ded1e255 not found: ID does not exist" containerID="da7156d8339ec9488e38599efafa17081e9925750125c778e6d950a7ded1e255" Oct 30 22:40:48.306552 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.306529 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da7156d8339ec9488e38599efafa17081e9925750125c778e6d950a7ded1e255"} err="failed to get container status \"da7156d8339ec9488e38599efafa17081e9925750125c778e6d950a7ded1e255\": rpc error: code = NotFound desc = could not find container \"da7156d8339ec9488e38599efafa17081e9925750125c778e6d950a7ded1e255\": container with ID starting with da7156d8339ec9488e38599efafa17081e9925750125c778e6d950a7ded1e255 not found: ID does not exist" Oct 30 22:40:48.306552 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.306552 6006 scope.go:122] "RemoveContainer" containerID="3f9f2b4ae0b8cee22c4c7b599a6def74ca28b91c76c68bcb6e74a1107238c824" Oct 30 22:40:48.319995 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.319867 6006 scope.go:122] "RemoveContainer" containerID="3f9f2b4ae0b8cee22c4c7b599a6def74ca28b91c76c68bcb6e74a1107238c824" Oct 30 22:40:48.320302 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:40:48.320269 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f9f2b4ae0b8cee22c4c7b599a6def74ca28b91c76c68bcb6e74a1107238c824\": container with ID starting with 3f9f2b4ae0b8cee22c4c7b599a6def74ca28b91c76c68bcb6e74a1107238c824 not found: ID does not exist" containerID="3f9f2b4ae0b8cee22c4c7b599a6def74ca28b91c76c68bcb6e74a1107238c824" Oct 30 22:40:48.320422 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.320316 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f9f2b4ae0b8cee22c4c7b599a6def74ca28b91c76c68bcb6e74a1107238c824"} err="failed to get container status \"3f9f2b4ae0b8cee22c4c7b599a6def74ca28b91c76c68bcb6e74a1107238c824\": rpc error: code = NotFound desc = could not find container \"3f9f2b4ae0b8cee22c4c7b599a6def74ca28b91c76c68bcb6e74a1107238c824\": container with ID starting with 3f9f2b4ae0b8cee22c4c7b599a6def74ca28b91c76c68bcb6e74a1107238c824 not found: ID does not exist" Oct 30 22:40:48.320422 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.320345 6006 scope.go:122] "RemoveContainer" containerID="c0d8465ee82e660a07575a51d1134dccdc1b9a394f57c5ee90613a704e83d89e" Oct 30 22:40:48.334658 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.333775 6006 scope.go:122] "RemoveContainer" containerID="c0d8465ee82e660a07575a51d1134dccdc1b9a394f57c5ee90613a704e83d89e" Oct 30 22:40:48.334658 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:40:48.334166 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0d8465ee82e660a07575a51d1134dccdc1b9a394f57c5ee90613a704e83d89e\": container with ID starting with c0d8465ee82e660a07575a51d1134dccdc1b9a394f57c5ee90613a704e83d89e not found: ID does not exist" containerID="c0d8465ee82e660a07575a51d1134dccdc1b9a394f57c5ee90613a704e83d89e" Oct 30 22:40:48.334658 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.334206 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0d8465ee82e660a07575a51d1134dccdc1b9a394f57c5ee90613a704e83d89e"} err="failed to get container status \"c0d8465ee82e660a07575a51d1134dccdc1b9a394f57c5ee90613a704e83d89e\": rpc error: code = NotFound desc = could not find container \"c0d8465ee82e660a07575a51d1134dccdc1b9a394f57c5ee90613a704e83d89e\": container with ID starting with c0d8465ee82e660a07575a51d1134dccdc1b9a394f57c5ee90613a704e83d89e not found: ID does not exist" Oct 30 22:40:48.416735 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.416649 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886"] Oct 30 22:40:48.439220 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.439183 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a00d9b/distinct-w1-a00d9b-kserve-85f69cb446-ln886"] Oct 30 22:40:48.624885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.624846 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn"] Oct 30 22:40:48.629166 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.629136 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a00d9b/premium-w1-a00d9b-kserve-77b5d48cf9-tz8fn"] Oct 30 22:40:48.660090 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.660053 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc"] Oct 30 22:40:48.663863 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.663836 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fb0b93-nr9zc"] Oct 30 22:40:48.732306 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.732237 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn"] Oct 30 22:40:48.739274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:48.739224 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn"] Oct 30 22:40:49.050894 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:49.050851 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a00d9b/simulator-w1-a00d9b-kserve-798c8dcf54-6h2gn" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 22:40:49.602401 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:49.602367 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5248e340-c665-485b-b2a6-0a07d14d5f53" path="/var/lib/kubelet/pods/5248e340-c665-485b-b2a6-0a07d14d5f53/volumes" Oct 30 22:40:49.602822 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:49.602807 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" path="/var/lib/kubelet/pods/8c71fda3-adf4-4b08-8a27-0065ac056a1d/volumes" Oct 30 22:40:49.603394 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:49.603375 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" path="/var/lib/kubelet/pods/b90b6498-0e99-4a90-b2ae-1bc4f2446d5b/volumes" Oct 30 22:40:49.604157 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:49.604141 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" path="/var/lib/kubelet/pods/d2715f84-b3fa-4d3d-80f9-18b5c8a7359a/volumes" Oct 30 22:40:53.134312 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:53.134274 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:40:56.704070 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:56.704031 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99" podUID="f9a2a7a5-468b-43f3-8528-1cc78b307324" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:40:57.216261 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:57.216220 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd"] Oct 30 22:40:57.216508 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:57.216483 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" podUID="ff455cf0-f681-43de-b6b8-b33eb3c40562" containerName="istio-proxy" containerID="cri-o://91e0f994561261eb2c2125848611ae1f727a40bc816f27d66a52dfec9f8fbe29" gracePeriod=30 Oct 30 22:40:57.911034 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:40:57.910992 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:40:59.783150 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:59.783116 6006 scope.go:122] "RemoveContainer" containerID="26ab1cd0d6d6f18b5340687e0ba32c70cbae8c07ef82d26c77d159ab09331c83" Oct 30 22:40:59.792549 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:59.792522 6006 scope.go:122] "RemoveContainer" containerID="0e8a83f72f41e90bbffb2130329ad438df892305fc0fda376d9d101c47973a3b" Oct 30 22:40:59.802206 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:40:59.802184 6006 scope.go:122] "RemoveContainer" containerID="431ad75dc4451a620bbbe589a7977f86df2eab14a3684253f5d3e237a87511df" Oct 30 22:41:02.410530 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.410492 6006 generic.go:356] "Generic (PLEG): container finished" podID="ff455cf0-f681-43de-b6b8-b33eb3c40562" containerID="91e0f994561261eb2c2125848611ae1f727a40bc816f27d66a52dfec9f8fbe29" exitCode=0 Oct 30 22:41:02.410934 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.410525 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" event={"ID":"ff455cf0-f681-43de-b6b8-b33eb3c40562","Type":"ContainerDied","Data":"91e0f994561261eb2c2125848611ae1f727a40bc816f27d66a52dfec9f8fbe29"} Oct 30 22:41:02.498805 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.498738 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:41:02.615104 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.615055 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff455cf0-f681-43de-b6b8-b33eb3c40562-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff455cf0-f681-43de-b6b8-b33eb3c40562-istiod-ca-cert\") pod \"ff455cf0-f681-43de-b6b8-b33eb3c40562\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " Oct 30 22:41:02.615104 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.615104 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-envoy\") pod \"ff455cf0-f681-43de-b6b8-b33eb3c40562\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " Oct 30 22:41:02.615433 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.615142 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-token\" (UniqueName: \"kubernetes.io/projected/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-token\") pod \"ff455cf0-f681-43de-b6b8-b33eb3c40562\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " Oct 30 22:41:02.615433 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.615185 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-data\") pod \"ff455cf0-f681-43de-b6b8-b33eb3c40562\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " Oct 30 22:41:02.615433 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.615210 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-workload-socket\") pod \"ff455cf0-f681-43de-b6b8-b33eb3c40562\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " Oct 30 22:41:02.615433 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.615293 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-credential-socket\") pod \"ff455cf0-f681-43de-b6b8-b33eb3c40562\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " Oct 30 22:41:02.615433 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.615334 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-workload-certs\") pod \"ff455cf0-f681-43de-b6b8-b33eb3c40562\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " Oct 30 22:41:02.615433 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.615397 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-podinfo\") pod \"ff455cf0-f681-43de-b6b8-b33eb3c40562\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " Oct 30 22:41:02.615433 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.615430 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff455cf0-f681-43de-b6b8-b33eb3c40562-kube-api-access-847lh\" (UniqueName: \"kubernetes.io/projected/ff455cf0-f681-43de-b6b8-b33eb3c40562-kube-api-access-847lh\") pod \"ff455cf0-f681-43de-b6b8-b33eb3c40562\" (UID: \"ff455cf0-f681-43de-b6b8-b33eb3c40562\") " Oct 30 22:41:02.615866 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.615592 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-data" pod "ff455cf0-f681-43de-b6b8-b33eb3c40562" (UID: "ff455cf0-f681-43de-b6b8-b33eb3c40562"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:02.615866 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.615684 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-workload-socket" pod "ff455cf0-f681-43de-b6b8-b33eb3c40562" (UID: "ff455cf0-f681-43de-b6b8-b33eb3c40562"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:02.615866 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.615736 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff455cf0-f681-43de-b6b8-b33eb3c40562-istiod-ca-cert" pod "ff455cf0-f681-43de-b6b8-b33eb3c40562" (UID: "ff455cf0-f681-43de-b6b8-b33eb3c40562"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:41:02.615866 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.615824 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-workload-certs" pod "ff455cf0-f681-43de-b6b8-b33eb3c40562" (UID: "ff455cf0-f681-43de-b6b8-b33eb3c40562"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:02.615866 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.615838 6006 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-data\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:02.615866 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.615860 6006 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-workload-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:02.616190 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.615877 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-credential-socket" pod "ff455cf0-f681-43de-b6b8-b33eb3c40562" (UID: "ff455cf0-f681-43de-b6b8-b33eb3c40562"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:02.617509 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.617485 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-envoy" pod "ff455cf0-f681-43de-b6b8-b33eb3c40562" (UID: "ff455cf0-f681-43de-b6b8-b33eb3c40562"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:02.617632 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.617576 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-token" pod "ff455cf0-f681-43de-b6b8-b33eb3c40562" (UID: "ff455cf0-f681-43de-b6b8-b33eb3c40562"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:02.617697 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.617666 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-podinfo" pod "ff455cf0-f681-43de-b6b8-b33eb3c40562" (UID: "ff455cf0-f681-43de-b6b8-b33eb3c40562"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:41:02.617795 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.617782 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff455cf0-f681-43de-b6b8-b33eb3c40562-kube-api-access-847lh" pod "ff455cf0-f681-43de-b6b8-b33eb3c40562" (UID: "ff455cf0-f681-43de-b6b8-b33eb3c40562"). InnerVolumeSpecName "kube-api-access-847lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:02.717273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.717210 6006 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff455cf0-f681-43de-b6b8-b33eb3c40562-istiod-ca-cert\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:02.717273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.717241 6006 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-envoy\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:02.717273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.717281 6006 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-token\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:02.717533 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.717291 6006 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-credential-socket\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:02.717533 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.717300 6006 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff455cf0-f681-43de-b6b8-b33eb3c40562-workload-certs\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:02.717533 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.717308 6006 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff455cf0-f681-43de-b6b8-b33eb3c40562-istio-podinfo\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:02.717533 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:02.717318 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-847lh\" (UniqueName: \"kubernetes.io/projected/ff455cf0-f681-43de-b6b8-b33eb3c40562-kube-api-access-847lh\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:03.417668 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:03.417628 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" event={"ID":"ff455cf0-f681-43de-b6b8-b33eb3c40562","Type":"ContainerDied","Data":"1ea7d68290b6b57c9b7a78ba9b80c0d1ec622c0273b4f4ffd640ff59c209b49d"} Oct 30 22:41:03.418088 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:03.417696 6006 scope.go:122] "RemoveContainer" containerID="91e0f994561261eb2c2125848611ae1f727a40bc816f27d66a52dfec9f8fbe29" Oct 30 22:41:03.418088 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:03.417638 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd" Oct 30 22:41:03.578068 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:03.578027 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd"] Oct 30 22:41:03.625155 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:03.625115 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb0b93-openshift-default-75ccb77bb5-7r4bd"] Oct 30 22:41:04.429441 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:04.429410 6006 generic.go:356] "Generic (PLEG): container finished" podID="f9a2a7a5-468b-43f3-8528-1cc78b307324" containerID="ea914c779525256ba30f6a32d4a6345f873203fb025f8c1e655dcc2e36fd40e4" exitCode=0 Oct 30 22:41:04.429857 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:04.429549 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99" event={"ID":"f9a2a7a5-468b-43f3-8528-1cc78b307324","Type":"ContainerDied","Data":"ea914c779525256ba30f6a32d4a6345f873203fb025f8c1e655dcc2e36fd40e4"} Oct 30 22:41:04.488342 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:04.488271 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99" Oct 30 22:41:04.637407 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:04.637359 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9a2a7a5-468b-43f3-8528-1cc78b307324-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a2a7a5-468b-43f3-8528-1cc78b307324-plugins-config-volume\") pod \"f9a2a7a5-468b-43f3-8528-1cc78b307324\" (UID: \"f9a2a7a5-468b-43f3-8528-1cc78b307324\") " Oct 30 22:41:04.637595 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:04.637551 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9a2a7a5-468b-43f3-8528-1cc78b307324-kube-api-access-lfc98\" (UniqueName: \"kubernetes.io/projected/f9a2a7a5-468b-43f3-8528-1cc78b307324-kube-api-access-lfc98\") pod \"f9a2a7a5-468b-43f3-8528-1cc78b307324\" (UID: \"f9a2a7a5-468b-43f3-8528-1cc78b307324\") " Oct 30 22:41:04.637779 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:04.637755 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9a2a7a5-468b-43f3-8528-1cc78b307324-plugins-config-volume" pod "f9a2a7a5-468b-43f3-8528-1cc78b307324" (UID: "f9a2a7a5-468b-43f3-8528-1cc78b307324"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:41:04.639799 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:04.639775 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a2a7a5-468b-43f3-8528-1cc78b307324-kube-api-access-lfc98" pod "f9a2a7a5-468b-43f3-8528-1cc78b307324" (UID: "f9a2a7a5-468b-43f3-8528-1cc78b307324"). InnerVolumeSpecName "kube-api-access-lfc98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:04.738911 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:04.738825 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfc98\" (UniqueName: \"kubernetes.io/projected/f9a2a7a5-468b-43f3-8528-1cc78b307324-kube-api-access-lfc98\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:04.738911 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:04.738857 6006 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a2a7a5-468b-43f3-8528-1cc78b307324-plugins-config-volume\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:05.436140 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:05.436092 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99" event={"ID":"f9a2a7a5-468b-43f3-8528-1cc78b307324","Type":"ContainerDied","Data":"c4532d66baf7cb5451437cf08cc714378fe1ae70cce6d2b887b2b85fe833250c"} Oct 30 22:41:05.436604 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:05.436168 6006 scope.go:122] "RemoveContainer" containerID="ea914c779525256ba30f6a32d4a6345f873203fb025f8c1e655dcc2e36fd40e4" Oct 30 22:41:05.436604 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:05.436116 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99" Oct 30 22:41:05.593237 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:05.593198 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99"] Oct 30 22:41:05.602687 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:05.602650 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff455cf0-f681-43de-b6b8-b33eb3c40562" path="/var/lib/kubelet/pods/ff455cf0-f681-43de-b6b8-b33eb3c40562/volumes" Oct 30 22:41:05.639807 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:05.639776 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a00d9b-c5759c9fc-qdx99"] Oct 30 22:41:07.602161 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:07.602128 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9a2a7a5-468b-43f3-8528-1cc78b307324" path="/var/lib/kubelet/pods/f9a2a7a5-468b-43f3-8528-1cc78b307324/volumes" Oct 30 22:41:13.547272 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:13.544674 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:41:27.912862 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:41:27.912820 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:41:40.910230 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:40.910185 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps"] Oct 30 22:41:40.910848 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:40.910783 6006 cpu_manager.go:414] "containerMap: removing container" podUID="a35ddaa8-6863-4c1f-b9e8-711a8e76e006" containerName="payload-processing" Oct 30 22:41:40.910848 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:40.910805 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a35ddaa8-6863-4c1f-b9e8-711a8e76e006" containerName="payload-processing" Oct 30 22:41:40.910848 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:40.910843 6006 cpu_manager.go:414] "containerMap: removing container" podUID="f9a2a7a5-468b-43f3-8528-1cc78b307324" containerName="payload-processing" Oct 30 22:41:40.910848 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:40.910850 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9a2a7a5-468b-43f3-8528-1cc78b307324" containerName="payload-processing" Oct 30 22:41:40.911073 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:40.910861 6006 cpu_manager.go:414] "containerMap: removing container" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" Oct 30 22:41:40.911073 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:40.910868 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" Oct 30 22:41:40.911073 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:40.910880 6006 cpu_manager.go:414] "containerMap: removing container" podUID="7891883f-2be5-4384-bdd3-861df80fd8fc" containerName="revoke-keys" Oct 30 22:41:40.911073 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:40.910888 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7891883f-2be5-4384-bdd3-861df80fd8fc" containerName="revoke-keys" Oct 30 22:41:40.911073 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:40.910909 6006 cpu_manager.go:414] "containerMap: removing container" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" Oct 30 22:41:40.911073 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:40.910917 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" Oct 30 22:41:40.911073 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:40.910935 6006 cpu_manager.go:414] "containerMap: removing container" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" Oct 30 22:41:40.911073 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:40.910942 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" Oct 30 22:41:40.911073 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:40.910957 6006 cpu_manager.go:414] "containerMap: removing container" podUID="ff455cf0-f681-43de-b6b8-b33eb3c40562" containerName="istio-proxy" Oct 30 22:41:40.911073 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:40.910965 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff455cf0-f681-43de-b6b8-b33eb3c40562" containerName="istio-proxy" Oct 30 22:41:40.911073 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:40.911056 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c71fda3-adf4-4b08-8a27-0065ac056a1d" containerName="main" Oct 30 22:41:40.911073 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:40.911067 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2715f84-b3fa-4d3d-80f9-18b5c8a7359a" containerName="main" Oct 30 22:41:40.911073 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:40.911076 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff455cf0-f681-43de-b6b8-b33eb3c40562" containerName="istio-proxy" Oct 30 22:41:40.911760 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:40.911089 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="a35ddaa8-6863-4c1f-b9e8-711a8e76e006" containerName="payload-processing" Oct 30 22:41:40.911760 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:40.911102 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9a2a7a5-468b-43f3-8528-1cc78b307324" containerName="payload-processing" Oct 30 22:41:40.911760 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:40.911110 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="b90b6498-0e99-4a90-b2ae-1bc4f2446d5b" containerName="main" Oct 30 22:41:40.911760 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:40.911119 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="7891883f-2be5-4384-bdd3-861df80fd8fc" containerName="revoke-keys" Oct 30 22:41:40.917237 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:40.917201 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:40.966465 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:40.966430 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v8cwr\"" Oct 30 22:41:41.029216 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.029179 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3b94ab-018e-480f-b12c-faa5544fc2e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.029463 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.029274 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c3b94ab-018e-480f-b12c-faa5544fc2e1-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.029463 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.029380 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3b94ab-018e-480f-b12c-faa5544fc2e1-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.029678 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.029473 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3b94ab-018e-480f-b12c-faa5544fc2e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.029678 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.029552 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3b94ab-018e-480f-b12c-faa5544fc2e1-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.029678 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.029586 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3b94ab-018e-480f-b12c-faa5544fc2e1-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.029885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.029689 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3b94ab-018e-480f-b12c-faa5544fc2e1-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.029885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.029719 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2cbj\" (UniqueName: \"kubernetes.io/projected/2c3b94ab-018e-480f-b12c-faa5544fc2e1-kube-api-access-b2cbj\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.029885 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.029771 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3b94ab-018e-480f-b12c-faa5544fc2e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.031216 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.031188 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps"] Oct 30 22:41:41.131046 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.131001 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3b94ab-018e-480f-b12c-faa5544fc2e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.131311 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.131059 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3b94ab-018e-480f-b12c-faa5544fc2e1-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.131311 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.131081 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3b94ab-018e-480f-b12c-faa5544fc2e1-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.131311 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.131119 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3b94ab-018e-480f-b12c-faa5544fc2e1-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.131311 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.131136 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2cbj\" (UniqueName: \"kubernetes.io/projected/2c3b94ab-018e-480f-b12c-faa5544fc2e1-kube-api-access-b2cbj\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.131311 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.131170 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3b94ab-018e-480f-b12c-faa5544fc2e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.131311 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.131196 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3b94ab-018e-480f-b12c-faa5544fc2e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.131311 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.131289 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c3b94ab-018e-480f-b12c-faa5544fc2e1-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.131706 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.131329 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3b94ab-018e-480f-b12c-faa5544fc2e1-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.131706 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.131539 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3b94ab-018e-480f-b12c-faa5544fc2e1-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.131706 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.131551 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3b94ab-018e-480f-b12c-faa5544fc2e1-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.131706 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.131619 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3b94ab-018e-480f-b12c-faa5544fc2e1-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.131935 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.131746 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3b94ab-018e-480f-b12c-faa5544fc2e1-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.132134 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.132106 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3b94ab-018e-480f-b12c-faa5544fc2e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.133515 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.133497 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3b94ab-018e-480f-b12c-faa5544fc2e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.134000 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.133976 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3b94ab-018e-480f-b12c-faa5544fc2e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.189728 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.189650 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2cbj\" (UniqueName: \"kubernetes.io/projected/2c3b94ab-018e-480f-b12c-faa5544fc2e1-kube-api-access-b2cbj\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.189728 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.189651 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c3b94ab-018e-480f-b12c-faa5544fc2e1-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6cmps\" (UID: \"2c3b94ab-018e-480f-b12c-faa5544fc2e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.232195 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.232148 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:41.654304 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.654267 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps"] Oct 30 22:41:41.656743 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:41:41.656707 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3b94ab_018e_480f_b12c_faa5544fc2e1.slice/crio-638554dfcc82e7f7a2cdf2f68e94bf0f1cc9b0bda3aec3cf90ace8be2f6b6ea6 WatchSource:0}: Error finding container 638554dfcc82e7f7a2cdf2f68e94bf0f1cc9b0bda3aec3cf90ace8be2f6b6ea6: Status 404 returned error can't find the container with id 638554dfcc82e7f7a2cdf2f68e94bf0f1cc9b0bda3aec3cf90ace8be2f6b6ea6 Oct 30 22:41:41.658883 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.658847 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:41:41.659000 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.658930 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:41:41.659000 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.658977 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:41:41.722476 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.722435 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" event={"ID":"2c3b94ab-018e-480f-b12c-faa5544fc2e1","Type":"ContainerStarted","Data":"ee2352530616d27cf14c9b37a3cb6a3be56898a0917a691614853548bb4de1bf"} Oct 30 22:41:41.722637 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.722500 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" event={"ID":"2c3b94ab-018e-480f-b12c-faa5544fc2e1","Type":"ContainerStarted","Data":"638554dfcc82e7f7a2cdf2f68e94bf0f1cc9b0bda3aec3cf90ace8be2f6b6ea6"} Oct 30 22:41:41.839157 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:41.839092 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" podStartSLOduration=1.839077386 podStartE2EDuration="1.839077386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:41.837452446 +0000 UTC m=+3104.389628747" watchObservedRunningTime="2026-10-30 22:41:41.839077386 +0000 UTC m=+3104.391253686" Oct 30 22:41:42.232404 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:42.232360 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:42.234122 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:42.234093 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-6cmps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 22:41:42.234297 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:42.234144 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" podUID="2c3b94ab-018e-480f-b12c-faa5544fc2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 22:41:43.233322 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:43.233285 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-6cmps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 22:41:43.233729 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:43.233344 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" podUID="2c3b94ab-018e-480f-b12c-faa5544fc2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 22:41:44.233591 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:44.233547 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-6cmps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 22:41:44.234045 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:44.233609 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" podUID="2c3b94ab-018e-480f-b12c-faa5544fc2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 22:41:45.233502 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:45.233463 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-6cmps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 22:41:45.233781 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:45.233522 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" podUID="2c3b94ab-018e-480f-b12c-faa5544fc2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 22:41:46.233320 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:46.233272 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-6cmps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 22:41:46.233530 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:46.233343 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" podUID="2c3b94ab-018e-480f-b12c-faa5544fc2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 22:41:47.232764 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:47.232721 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-6cmps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 22:41:47.233196 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:47.232782 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" podUID="2c3b94ab-018e-480f-b12c-faa5544fc2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 22:41:48.232961 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:48.232921 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-6cmps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 22:41:48.233375 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:48.232978 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" podUID="2c3b94ab-018e-480f-b12c-faa5544fc2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 22:41:49.938597 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:49.938547 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:49.939120 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:49.939094 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:49.939990 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:41:49.939965 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" Oct 30 22:41:57.908827 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:41:57.908784 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:42:27.054443 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:27.054403 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48bd57a2-8hvpj"] Oct 30 22:42:27.090895 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:27.090852 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48bd57a2-8hvpj"] Oct 30 22:42:27.602584 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:27.602553 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71d8d509-855b-4e9d-b9a0-8b62fbb40944" path="/var/lib/kubelet/pods/71d8d509-855b-4e9d-b9a0-8b62fbb40944/volumes" Oct 30 22:42:27.909994 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:42:27.909913 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:42:32.610573 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:32.610528 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:42:40.052929 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:40.052837 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-225aacce-vfsft"] Oct 30 22:42:40.100360 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:40.100321 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-225aacce-vfsft"] Oct 30 22:42:41.602848 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:41.602812 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3a268ec-2276-420c-8533-5a9d302381b1" path="/var/lib/kubelet/pods/e3a268ec-2276-420c-8533-5a9d302381b1/volumes" Oct 30 22:42:53.930280 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:53.930224 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:42:56.491120 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.491077 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g"] Oct 30 22:42:56.494918 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.494893 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.546757 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.546712 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a12523-b466-4041-bb2a-73db5538706c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.546960 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.546774 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a12523-b466-4041-bb2a-73db5538706c-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.546960 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.546893 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a12523-b466-4041-bb2a-73db5538706c-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.546960 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.546937 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a12523-b466-4041-bb2a-73db5538706c-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.547109 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.546963 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a12523-b466-4041-bb2a-73db5538706c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.547109 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.547076 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8jz5\" (UniqueName: \"kubernetes.io/projected/82a12523-b466-4041-bb2a-73db5538706c-kube-api-access-k8jz5\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.547208 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.547108 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a12523-b466-4041-bb2a-73db5538706c-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.547208 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.547152 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a12523-b466-4041-bb2a-73db5538706c-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.547330 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.547214 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a12523-b466-4041-bb2a-73db5538706c-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.647993 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.647955 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a12523-b466-4041-bb2a-73db5538706c-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.647993 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.647998 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a12523-b466-4041-bb2a-73db5538706c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.648233 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.648030 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8jz5\" (UniqueName: \"kubernetes.io/projected/82a12523-b466-4041-bb2a-73db5538706c-kube-api-access-k8jz5\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.648233 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.648049 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a12523-b466-4041-bb2a-73db5538706c-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.648233 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.648097 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a12523-b466-4041-bb2a-73db5538706c-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.648233 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.648121 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a12523-b466-4041-bb2a-73db5538706c-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.648233 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.648166 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a12523-b466-4041-bb2a-73db5538706c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.648233 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.648199 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a12523-b466-4041-bb2a-73db5538706c-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.648233 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.648220 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a12523-b466-4041-bb2a-73db5538706c-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.649140 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.649104 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a12523-b466-4041-bb2a-73db5538706c-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.649302 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.649161 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a12523-b466-4041-bb2a-73db5538706c-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.649302 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.649213 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a12523-b466-4041-bb2a-73db5538706c-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.649438 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.649364 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a12523-b466-4041-bb2a-73db5538706c-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.649549 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.649524 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a12523-b466-4041-bb2a-73db5538706c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.651165 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.651143 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a12523-b466-4041-bb2a-73db5538706c-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.651312 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.651192 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a12523-b466-4041-bb2a-73db5538706c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.687710 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.687675 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g"] Oct 30 22:42:56.706702 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.706673 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a12523-b466-4041-bb2a-73db5538706c-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.706863 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.706708 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8jz5\" (UniqueName: \"kubernetes.io/projected/82a12523-b466-4041-bb2a-73db5538706c-kube-api-access-k8jz5\") pod \"maas-default-gateway-openshift-default-85544d4cd7-jst4g\" (UID: \"82a12523-b466-4041-bb2a-73db5538706c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:56.809074 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:56.809037 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:57.230901 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:57.230850 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g"] Oct 30 22:42:57.233687 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:42:57.233655 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a12523_b466_4041_bb2a_73db5538706c.slice/crio-257b5dfa5799d35e86a2f9eb67f3b5f5a3c8fa236e095bd3e0486c665a7f5313 WatchSource:0}: Error finding container 257b5dfa5799d35e86a2f9eb67f3b5f5a3c8fa236e095bd3e0486c665a7f5313: Status 404 returned error can't find the container with id 257b5dfa5799d35e86a2f9eb67f3b5f5a3c8fa236e095bd3e0486c665a7f5313 Oct 30 22:42:57.236287 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:57.235990 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:42:57.236287 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:57.236079 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:42:57.236287 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:57.236128 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:42:57.933899 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:42:57.933862 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:42:58.145446 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:58.145408 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" event={"ID":"82a12523-b466-4041-bb2a-73db5538706c","Type":"ContainerStarted","Data":"ed98e63a8193bf62a4d1ffaec3198d0becfbdf52ec40492627086d89796144b7"} Oct 30 22:42:58.145650 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:58.145477 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" event={"ID":"82a12523-b466-4041-bb2a-73db5538706c","Type":"ContainerStarted","Data":"257b5dfa5799d35e86a2f9eb67f3b5f5a3c8fa236e095bd3e0486c665a7f5313"} Oct 30 22:42:58.357075 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:58.357004 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" podStartSLOduration=2.356992006 podStartE2EDuration="2.356992006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:58.318572028 +0000 UTC m=+3180.870748336" watchObservedRunningTime="2026-10-30 22:42:58.356992006 +0000 UTC m=+3180.909168305" Oct 30 22:42:58.819274 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:58.818316 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:42:58.823272 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:58.820366 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-jst4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 30 22:42:58.823272 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:58.820426 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" podUID="82a12523-b466-4041-bb2a-73db5538706c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 30 22:42:59.812770 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:59.812723 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-jst4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 30 22:42:59.813460 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:42:59.813425 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" podUID="82a12523-b466-4041-bb2a-73db5538706c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 30 22:43:00.809781 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:00.809739 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-jst4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 30 22:43:00.810143 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:00.810106 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" podUID="82a12523-b466-4041-bb2a-73db5538706c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 30 22:43:01.809778 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:01.809732 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-jst4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 30 22:43:01.810475 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:01.810440 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" podUID="82a12523-b466-4041-bb2a-73db5538706c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 30 22:43:02.811784 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:02.811739 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-jst4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 30 22:43:02.812448 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:02.812413 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" podUID="82a12523-b466-4041-bb2a-73db5538706c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 30 22:43:03.809602 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:03.809561 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-jst4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 30 22:43:03.809800 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:03.809619 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" podUID="82a12523-b466-4041-bb2a-73db5538706c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 30 22:43:04.810404 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:04.810353 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-jst4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 30 22:43:04.810850 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:04.810448 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" podUID="82a12523-b466-4041-bb2a-73db5538706c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 30 22:43:06.331224 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:06.331185 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:43:06.850516 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:06.850462 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-jst4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:43:06.850765 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:06.850567 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" podUID="82a12523-b466-4041-bb2a-73db5538706c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:43:07.765843 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:07.765809 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:43:07.766581 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:07.766563 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:43:07.767428 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:07.767408 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-jst4g" Oct 30 22:43:20.048382 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:20.048341 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f053c1c1-hxg8j"] Oct 30 22:43:20.093864 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:20.093827 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f053c1c1-hxg8j"] Oct 30 22:43:21.602282 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:21.602218 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27986857-6552-4518-bc63-da753984fc2e" path="/var/lib/kubelet/pods/27986857-6552-4518-bc63-da753984fc2e/volumes" Oct 30 22:43:25.416658 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:25.416617 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:43:26.719711 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.719673 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj"] Oct 30 22:43:26.723564 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.723536 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:26.823354 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.823312 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj"] Oct 30 22:43:26.865801 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.865754 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b53342-a5ab-459b-8005-da1267a7c0b4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:26.866028 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.865869 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b53342-a5ab-459b-8005-da1267a7c0b4-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:26.866028 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.865928 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b53342-a5ab-459b-8005-da1267a7c0b4-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:26.866028 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.866006 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b53342-a5ab-459b-8005-da1267a7c0b4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:26.866308 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.866113 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b53342-a5ab-459b-8005-da1267a7c0b4-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:26.866308 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.866164 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b53342-a5ab-459b-8005-da1267a7c0b4-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:26.866308 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.866196 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b53342-a5ab-459b-8005-da1267a7c0b4-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:26.866308 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.866223 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b53342-a5ab-459b-8005-da1267a7c0b4-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:26.866538 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.866340 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6hb9\" (UniqueName: \"kubernetes.io/projected/33b53342-a5ab-459b-8005-da1267a7c0b4-kube-api-access-v6hb9\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:26.967666 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.967623 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6hb9\" (UniqueName: \"kubernetes.io/projected/33b53342-a5ab-459b-8005-da1267a7c0b4-kube-api-access-v6hb9\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:26.967666 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.967670 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b53342-a5ab-459b-8005-da1267a7c0b4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:26.967956 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.967723 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b53342-a5ab-459b-8005-da1267a7c0b4-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:26.967956 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.967773 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b53342-a5ab-459b-8005-da1267a7c0b4-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:26.967956 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.967796 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b53342-a5ab-459b-8005-da1267a7c0b4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:26.967956 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.967835 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b53342-a5ab-459b-8005-da1267a7c0b4-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:26.967956 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.967858 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b53342-a5ab-459b-8005-da1267a7c0b4-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:26.967956 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.967883 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b53342-a5ab-459b-8005-da1267a7c0b4-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:26.967956 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.967915 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b53342-a5ab-459b-8005-da1267a7c0b4-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:26.968385 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.968241 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b53342-a5ab-459b-8005-da1267a7c0b4-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:26.968385 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.968342 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b53342-a5ab-459b-8005-da1267a7c0b4-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:26.968511 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.968445 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b53342-a5ab-459b-8005-da1267a7c0b4-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:26.968511 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.968479 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b53342-a5ab-459b-8005-da1267a7c0b4-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:26.968732 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.968704 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b53342-a5ab-459b-8005-da1267a7c0b4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:26.970895 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.970843 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b53342-a5ab-459b-8005-da1267a7c0b4-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:26.971015 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:26.970996 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b53342-a5ab-459b-8005-da1267a7c0b4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:27.026994 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:27.026957 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b53342-a5ab-459b-8005-da1267a7c0b4-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:27.027185 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:27.027159 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6hb9\" (UniqueName: \"kubernetes.io/projected/33b53342-a5ab-459b-8005-da1267a7c0b4-kube-api-access-v6hb9\") pod \"maas-default-gateway-openshift-default-85544d4cd7-ck8bj\" (UID: \"33b53342-a5ab-459b-8005-da1267a7c0b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:27.040006 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:27.039898 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:27.473237 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:27.473208 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj"] Oct 30 22:43:27.475832 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:43:27.475800 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b53342_a5ab_459b_8005_da1267a7c0b4.slice/crio-e49780238e8f0ff9887062af1935bef02a711f8c3c301b81788f203998dde2ae WatchSource:0}: Error finding container e49780238e8f0ff9887062af1935bef02a711f8c3c301b81788f203998dde2ae: Status 404 returned error can't find the container with id e49780238e8f0ff9887062af1935bef02a711f8c3c301b81788f203998dde2ae Oct 30 22:43:27.477941 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:27.477908 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:43:27.477999 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:27.477979 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:43:27.478038 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:27.478006 6006 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 22:43:27.920577 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:43:27.917025 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:43:28.317960 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:28.317918 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" event={"ID":"33b53342-a5ab-459b-8005-da1267a7c0b4","Type":"ContainerStarted","Data":"7424816923a11b7a23d6269e5361611d52ea174be7a32bc76d60f4f5d253cce9"} Oct 30 22:43:28.319391 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:28.319357 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" event={"ID":"33b53342-a5ab-459b-8005-da1267a7c0b4","Type":"ContainerStarted","Data":"e49780238e8f0ff9887062af1935bef02a711f8c3c301b81788f203998dde2ae"} Oct 30 22:43:28.489780 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:28.489714 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" podStartSLOduration=2.489696652 podStartE2EDuration="2.489696652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:28.483675066 +0000 UTC m=+3211.035851366" watchObservedRunningTime="2026-10-30 22:43:28.489696652 +0000 UTC m=+3211.041872953" Oct 30 22:43:29.041835 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:29.041784 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:29.043618 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:29.043590 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-ck8bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 22:43:29.043743 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:29.043642 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" podUID="33b53342-a5ab-459b-8005-da1267a7c0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 22:43:30.041908 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:30.041855 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-ck8bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 22:43:30.042359 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:30.041949 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" podUID="33b53342-a5ab-459b-8005-da1267a7c0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 22:43:31.041149 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:31.041110 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-ck8bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 22:43:31.041396 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:31.041171 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" podUID="33b53342-a5ab-459b-8005-da1267a7c0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 22:43:32.041812 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:32.041771 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-ck8bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 22:43:32.042215 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:32.041848 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" podUID="33b53342-a5ab-459b-8005-da1267a7c0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 22:43:33.041553 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:33.041515 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-ck8bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 22:43:33.041756 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:33.041573 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" podUID="33b53342-a5ab-459b-8005-da1267a7c0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 22:43:34.041164 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:34.041122 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-ck8bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 22:43:34.044782 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:34.041193 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" podUID="33b53342-a5ab-459b-8005-da1267a7c0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 22:43:35.041677 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:35.041633 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-ck8bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 22:43:35.042181 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:35.041686 6006 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" podUID="33b53342-a5ab-459b-8005-da1267a7c0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 22:43:36.046423 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:36.046390 6006 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:36.046898 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:36.046758 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:36.047798 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:36.047778 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" Oct 30 22:43:42.409765 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:42.409727 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:43:44.067272 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:44.066447 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32401d4a-6xph4"] Oct 30 22:43:44.093076 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:44.093023 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32401d4a-6xph4"] Oct 30 22:43:45.602983 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:45.602945 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03b0d538-05a6-4897-aae0-85f5ccc21dd0" path="/var/lib/kubelet/pods/03b0d538-05a6-4897-aae0-85f5ccc21dd0/volumes" Oct 30 22:43:57.910768 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:43:57.910720 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:43:59.878831 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:59.878798 6006 scope.go:122] "RemoveContainer" containerID="d1dee38f3771b6656d4ada3f72bed3ddb540ebd0566a31da8b238801ee8c41ff" Oct 30 22:43:59.888204 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:43:59.888181 6006 scope.go:122] "RemoveContainer" containerID="76becb12bf7e19163939d22d7834a6f9ae6601c1160ae0de3e98d48bc4ad1c8c" Oct 30 22:44:13.126484 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:44:13.126398 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:44:23.721261 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:44:23.721222 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:44:27.909825 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:44:27.909789 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:44:34.408711 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:44:34.408663 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:44:57.911735 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:44:57.911677 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:44:58.181152 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:44:58.181071 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 22:44:58.199317 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:44:58.199280 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 22:44:58.693551 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:44:58.693508 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:44:59.918347 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:44:59.918312 6006 scope.go:122] "RemoveContainer" containerID="3b40aa92ce424e673d8adc31f118f823699c6130696560531e3c8f40671e2d6a" Oct 30 22:44:59.928488 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:44:59.928465 6006 scope.go:122] "RemoveContainer" containerID="473399c58f79774683ea2fba30265ade1cc53c52414f43b7d315b9ac515576af" Oct 30 22:44:59.937561 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:44:59.937539 6006 scope.go:122] "RemoveContainer" containerID="b5d6d6b1e80f937c6429b7f3037f309f2497b9c41b2224ad49432e578e7b669c" Oct 30 22:44:59.947023 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:44:59.947004 6006 scope.go:122] "RemoveContainer" containerID="e878eb9184c8a7bbc1c91e09a44e81ad6c2b1d5e9124f2eb3b0487939ad2f6a0" Oct 30 22:45:05.984478 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:05.984424 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-6cmps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:45:05.985180 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:05.984531 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" podUID="2c3b94ab-018e-480f-b12c-faa5544fc2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:45:08.099191 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:08.099154 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:45:14.166953 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:14.166911 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6ckkm"] Oct 30 22:45:14.172377 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:14.172354 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6ckkm" Oct 30 22:45:14.226821 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:14.226782 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6ckkm"] Oct 30 22:45:14.271134 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:14.271093 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:45:14.271346 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:14.271144 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:45:14.271346 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:14.271180 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:45:14.271346 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:14.271202 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:45:14.277880 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:14.277847 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d8abf92-a63c-4368-aaf7-a2008135702b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6ckkm\" (UID: \"2d8abf92-a63c-4368-aaf7-a2008135702b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6ckkm" Oct 30 22:45:14.278036 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:14.277906 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxxw4\" (UniqueName: \"kubernetes.io/projected/2d8abf92-a63c-4368-aaf7-a2008135702b-kube-api-access-kxxw4\") pod \"maas-api-revoke-sub-a4aef7d7-6ckkm\" (UID: \"2d8abf92-a63c-4368-aaf7-a2008135702b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6ckkm" Oct 30 22:45:14.278089 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:14.278043 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d8abf92-a63c-4368-aaf7-a2008135702b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6ckkm\" (UID: \"2d8abf92-a63c-4368-aaf7-a2008135702b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6ckkm" Oct 30 22:45:14.379265 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:14.379209 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d8abf92-a63c-4368-aaf7-a2008135702b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6ckkm\" (UID: \"2d8abf92-a63c-4368-aaf7-a2008135702b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6ckkm" Oct 30 22:45:14.379488 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:14.379295 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxxw4\" (UniqueName: \"kubernetes.io/projected/2d8abf92-a63c-4368-aaf7-a2008135702b-kube-api-access-kxxw4\") pod \"maas-api-revoke-sub-a4aef7d7-6ckkm\" (UID: \"2d8abf92-a63c-4368-aaf7-a2008135702b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6ckkm" Oct 30 22:45:14.379488 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:14.379338 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d8abf92-a63c-4368-aaf7-a2008135702b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6ckkm\" (UID: \"2d8abf92-a63c-4368-aaf7-a2008135702b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6ckkm" Oct 30 22:45:14.380014 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:14.379989 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d8abf92-a63c-4368-aaf7-a2008135702b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6ckkm\" (UID: \"2d8abf92-a63c-4368-aaf7-a2008135702b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6ckkm" Oct 30 22:45:14.381672 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:14.381647 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d8abf92-a63c-4368-aaf7-a2008135702b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6ckkm\" (UID: \"2d8abf92-a63c-4368-aaf7-a2008135702b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6ckkm" Oct 30 22:45:14.438577 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:14.438503 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxxw4\" (UniqueName: \"kubernetes.io/projected/2d8abf92-a63c-4368-aaf7-a2008135702b-kube-api-access-kxxw4\") pod \"maas-api-revoke-sub-a4aef7d7-6ckkm\" (UID: \"2d8abf92-a63c-4368-aaf7-a2008135702b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6ckkm" Oct 30 22:45:14.485365 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:14.485332 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6ckkm" Oct 30 22:45:14.860432 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:14.860407 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6ckkm"] Oct 30 22:45:14.862929 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:45:14.862892 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d8abf92_a63c_4368_aaf7_a2008135702b.slice/crio-f5f4c666f86ec504875246b6ee5dbe1f15398011e92f4e5eb605d5ffb3d60869 WatchSource:0}: Error finding container f5f4c666f86ec504875246b6ee5dbe1f15398011e92f4e5eb605d5ffb3d60869: Status 404 returned error can't find the container with id f5f4c666f86ec504875246b6ee5dbe1f15398011e92f4e5eb605d5ffb3d60869 Oct 30 22:45:15.029823 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:15.029786 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6ckkm" event={"ID":"2d8abf92-a63c-4368-aaf7-a2008135702b","Type":"ContainerStarted","Data":"f5f4c666f86ec504875246b6ee5dbe1f15398011e92f4e5eb605d5ffb3d60869"} Oct 30 22:45:16.037124 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:16.037027 6006 generic.go:356] "Generic (PLEG): container finished" podID="2d8abf92-a63c-4368-aaf7-a2008135702b" containerID="a9608efcb311cf1ff9637115e93938c91095fd965e1fdcb338d9d97819f358f9" exitCode=0 Oct 30 22:45:16.037555 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:16.037116 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6ckkm" event={"ID":"2d8abf92-a63c-4368-aaf7-a2008135702b","Type":"ContainerDied","Data":"a9608efcb311cf1ff9637115e93938c91095fd965e1fdcb338d9d97819f358f9"} Oct 30 22:45:17.186960 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:17.186931 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6ckkm" Oct 30 22:45:17.308969 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:17.308882 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d8abf92-a63c-4368-aaf7-a2008135702b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d8abf92-a63c-4368-aaf7-a2008135702b-maas-api-xks-ca\") pod \"2d8abf92-a63c-4368-aaf7-a2008135702b\" (UID: \"2d8abf92-a63c-4368-aaf7-a2008135702b\") " Oct 30 22:45:17.309151 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:17.309104 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d8abf92-a63c-4368-aaf7-a2008135702b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d8abf92-a63c-4368-aaf7-a2008135702b-maas-api-openshift-ca\") pod \"2d8abf92-a63c-4368-aaf7-a2008135702b\" (UID: \"2d8abf92-a63c-4368-aaf7-a2008135702b\") " Oct 30 22:45:17.309237 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:17.309164 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d8abf92-a63c-4368-aaf7-a2008135702b-kube-api-access-kxxw4\" (UniqueName: \"kubernetes.io/projected/2d8abf92-a63c-4368-aaf7-a2008135702b-kube-api-access-kxxw4\") pod \"2d8abf92-a63c-4368-aaf7-a2008135702b\" (UID: \"2d8abf92-a63c-4368-aaf7-a2008135702b\") " Oct 30 22:45:17.309527 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:17.309501 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d8abf92-a63c-4368-aaf7-a2008135702b-maas-api-openshift-ca" pod "2d8abf92-a63c-4368-aaf7-a2008135702b" (UID: "2d8abf92-a63c-4368-aaf7-a2008135702b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:17.311307 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:17.311280 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8abf92-a63c-4368-aaf7-a2008135702b-kube-api-access-kxxw4" pod "2d8abf92-a63c-4368-aaf7-a2008135702b" (UID: "2d8abf92-a63c-4368-aaf7-a2008135702b"). InnerVolumeSpecName "kube-api-access-kxxw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:17.311580 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:17.311562 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d8abf92-a63c-4368-aaf7-a2008135702b-maas-api-xks-ca" pod "2d8abf92-a63c-4368-aaf7-a2008135702b" (UID: "2d8abf92-a63c-4368-aaf7-a2008135702b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:45:17.410816 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:17.410773 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d8abf92-a63c-4368-aaf7-a2008135702b-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:17.410816 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:17.410809 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxxw4\" (UniqueName: \"kubernetes.io/projected/2d8abf92-a63c-4368-aaf7-a2008135702b-kube-api-access-kxxw4\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:17.410816 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:17.410823 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d8abf92-a63c-4368-aaf7-a2008135702b-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:18.049167 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:18.049137 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6ckkm" Oct 30 22:45:18.049368 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:18.049174 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6ckkm" event={"ID":"2d8abf92-a63c-4368-aaf7-a2008135702b","Type":"ContainerDied","Data":"f5f4c666f86ec504875246b6ee5dbe1f15398011e92f4e5eb605d5ffb3d60869"} Oct 30 22:45:18.049368 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:18.049223 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5f4c666f86ec504875246b6ee5dbe1f15398011e92f4e5eb605d5ffb3d60869" Oct 30 22:45:26.941888 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:26.941853 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:45:27.911424 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:45:27.911382 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:45:35.985496 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:35.985395 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-6cmps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:45:35.985496 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:35.985476 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" podUID="2c3b94ab-018e-480f-b12c-faa5544fc2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:45:36.056276 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:36.055317 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10e61ea0-22k8q"] Oct 30 22:45:36.092283 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:36.092229 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10e61ea0-22k8q"] Oct 30 22:45:37.602841 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:37.602806 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b1283cb-bd03-435b-b957-0721eb8995c4" path="/var/lib/kubelet/pods/9b1283cb-bd03-435b-b957-0721eb8995c4/volumes" Oct 30 22:45:39.109342 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:39.109289 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:45:46.048816 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:46.048778 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-869d3806-cvr54"] Oct 30 22:45:46.095496 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:46.095459 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-869d3806-cvr54"] Oct 30 22:45:47.602943 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:47.602906 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7891883f-2be5-4384-bdd3-861df80fd8fc" path="/var/lib/kubelet/pods/7891883f-2be5-4384-bdd3-861df80fd8fc/volumes" Oct 30 22:45:57.912439 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:45:57.912399 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:45:59.430136 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:59.430095 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:45:59.984289 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:59.984233 6006 scope.go:122] "RemoveContainer" containerID="bae079c7c3bf8302fda12b786f5f98f5f2946cc53cc664c73c1d45834e6461e5" Oct 30 22:45:59.996051 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:45:59.996023 6006 scope.go:122] "RemoveContainer" containerID="2801ab17f7c3e15202450d85e720c3f913996f6b2ff72ad5cfc965bd99bff439" Oct 30 22:46:00.023406 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:46:00.023234 6006 scope.go:122] "RemoveContainer" containerID="9c4092cd72e1af63cfaed71a49cdc1c1ed718b8cf89632a1c503dd433e00b504" Oct 30 22:46:21.700088 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:46:21.700040 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:46:27.910706 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:46:27.910662 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:46:34.548429 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:46:34.548392 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:46:37.096474 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:46:37.096437 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-ck8bj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:46:37.096947 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:46:37.096527 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" podUID="33b53342-a5ab-459b-8005-da1267a7c0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:46:45.331689 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:46:45.331639 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:46:57.910232 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:46:57.910187 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:47:00.066155 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:47:00.066120 6006 scope.go:122] "RemoveContainer" containerID="5b4839d67c0ecb9a8a3621ef7f4a7496dce5e956facffcda44a1edb821de5a6b" Oct 30 22:47:00.075365 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:47:00.075343 6006 scope.go:122] "RemoveContainer" containerID="771b30343eb77bf997e51a3a212d2c96b54104b4334fa886480a3fc50f9a7fa5" Oct 30 22:47:00.084529 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:47:00.084501 6006 scope.go:122] "RemoveContainer" containerID="d304753395bd9cb2cf5db23a1dbfc681f0f6798e9674665a9f1fc4e6f332295c" Oct 30 22:47:00.093658 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:47:00.093636 6006 scope.go:122] "RemoveContainer" containerID="f59d90a61649c0e533739321b8dc6e3461dea4043041fce4bf6c226b157ddc34" Oct 30 22:47:05.292944 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:47:05.292907 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:47:15.131025 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:47:15.130985 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:47:27.910107 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:47:27.910067 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:47:30.809349 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:47:30.809308 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:47:54.086521 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:47:54.086477 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:47:57.911547 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:47:57.911498 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:48:04.812489 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:48:04.812449 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:48:27.095946 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:48:27.095898 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:48:27.988492 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:48:27.988415 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:48:37.319101 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:48:37.318996 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:48:57.911375 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:48:57.911321 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:49:03.427256 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:49:03.427191 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:49:22.348273 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:49:22.342929 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:49:27.910858 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:49:27.910817 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:49:42.936317 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:49:42.934530 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:49:57.910265 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:49:57.910203 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:49:58.212606 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:49:58.212526 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 22:49:58.246565 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:49:58.246535 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 22:50:12.196214 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:50:12.196121 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:50:18.063007 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:50:18.062969 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6ckkm"] Oct 30 22:50:18.107728 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:50:18.107694 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6ckkm"] Oct 30 22:50:19.602319 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:50:19.602242 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d8abf92-a63c-4368-aaf7-a2008135702b" path="/var/lib/kubelet/pods/2d8abf92-a63c-4368-aaf7-a2008135702b/volumes" Oct 30 22:50:27.910853 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:50:27.910810 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:50:29.626440 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:50:29.626396 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:50:50.416263 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:50:50.416219 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:50:52.092458 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:50:52.092401 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-ck8bj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:50:52.092908 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:50:52.092496 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-ck8bj" podUID="33b53342-a5ab-459b-8005-da1267a7c0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:50:57.910067 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:50:57.910029 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:51:00.940936 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:51:00.940897 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:51:10.424672 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:51:10.424633 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4hvz4"] Oct 30 22:51:10.425093 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:51:10.424865 6006 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4hvz4" podUID="297029e1-47c7-4aca-8699-4d94f72cd104" containerName="manager" containerID="cri-o://6fbd3a5356dc52d067bf4726fc94d9c090ee4a5a6d489cce674c93c79f6bb0b9" gracePeriod=10 Oct 30 22:51:10.693761 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:51:10.693735 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4hvz4" Oct 30 22:51:10.793893 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:51:10.793852 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297029e1-47c7-4aca-8699-4d94f72cd104-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/297029e1-47c7-4aca-8699-4d94f72cd104-extensions-socket-volume\") pod \"297029e1-47c7-4aca-8699-4d94f72cd104\" (UID: \"297029e1-47c7-4aca-8699-4d94f72cd104\") " Oct 30 22:51:10.794080 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:51:10.793930 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/297029e1-47c7-4aca-8699-4d94f72cd104-kube-api-access-4ldjj\" (UniqueName: \"kubernetes.io/projected/297029e1-47c7-4aca-8699-4d94f72cd104-kube-api-access-4ldjj\") pod \"297029e1-47c7-4aca-8699-4d94f72cd104\" (UID: \"297029e1-47c7-4aca-8699-4d94f72cd104\") " Oct 30 22:51:10.794367 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:51:10.794232 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297029e1-47c7-4aca-8699-4d94f72cd104-extensions-socket-volume" pod "297029e1-47c7-4aca-8699-4d94f72cd104" (UID: "297029e1-47c7-4aca-8699-4d94f72cd104"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:51:10.796367 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:51:10.796343 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/297029e1-47c7-4aca-8699-4d94f72cd104-kube-api-access-4ldjj" pod "297029e1-47c7-4aca-8699-4d94f72cd104" (UID: "297029e1-47c7-4aca-8699-4d94f72cd104"). InnerVolumeSpecName "kube-api-access-4ldjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:51:10.895455 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:51:10.895413 6006 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/297029e1-47c7-4aca-8699-4d94f72cd104-extensions-socket-volume\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:10.895619 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:51:10.895459 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ldjj\" (UniqueName: \"kubernetes.io/projected/297029e1-47c7-4aca-8699-4d94f72cd104-kube-api-access-4ldjj\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:11.123940 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:51:11.123895 6006 generic.go:356] "Generic (PLEG): container finished" podID="297029e1-47c7-4aca-8699-4d94f72cd104" containerID="6fbd3a5356dc52d067bf4726fc94d9c090ee4a5a6d489cce674c93c79f6bb0b9" exitCode=0 Oct 30 22:51:11.123940 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:51:11.123939 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4hvz4" Oct 30 22:51:11.124199 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:51:11.123962 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4hvz4" event={"ID":"297029e1-47c7-4aca-8699-4d94f72cd104","Type":"ContainerDied","Data":"6fbd3a5356dc52d067bf4726fc94d9c090ee4a5a6d489cce674c93c79f6bb0b9"} Oct 30 22:51:11.124199 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:51:11.124009 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4hvz4" event={"ID":"297029e1-47c7-4aca-8699-4d94f72cd104","Type":"ContainerDied","Data":"9e48fb2dff6f4505ca6ea44d9485d2ecbc7b74a5c36b3dc0a1518ba504906100"} Oct 30 22:51:11.124199 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:51:11.124034 6006 scope.go:122] "RemoveContainer" containerID="6fbd3a5356dc52d067bf4726fc94d9c090ee4a5a6d489cce674c93c79f6bb0b9" Oct 30 22:51:11.135633 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:51:11.135613 6006 scope.go:122] "RemoveContainer" containerID="6fbd3a5356dc52d067bf4726fc94d9c090ee4a5a6d489cce674c93c79f6bb0b9" Oct 30 22:51:11.135920 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:51:11.135896 6006 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fbd3a5356dc52d067bf4726fc94d9c090ee4a5a6d489cce674c93c79f6bb0b9\": container with ID starting with 6fbd3a5356dc52d067bf4726fc94d9c090ee4a5a6d489cce674c93c79f6bb0b9 not found: ID does not exist" containerID="6fbd3a5356dc52d067bf4726fc94d9c090ee4a5a6d489cce674c93c79f6bb0b9" Oct 30 22:51:11.135978 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:51:11.135929 6006 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fbd3a5356dc52d067bf4726fc94d9c090ee4a5a6d489cce674c93c79f6bb0b9"} err="failed to get container status \"6fbd3a5356dc52d067bf4726fc94d9c090ee4a5a6d489cce674c93c79f6bb0b9\": rpc error: code = NotFound desc = could not find container \"6fbd3a5356dc52d067bf4726fc94d9c090ee4a5a6d489cce674c93c79f6bb0b9\": container with ID starting with 6fbd3a5356dc52d067bf4726fc94d9c090ee4a5a6d489cce674c93c79f6bb0b9 not found: ID does not exist" Oct 30 22:51:11.278893 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:51:11.278854 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4hvz4"] Oct 30 22:51:11.284812 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:51:11.284783 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4hvz4"] Oct 30 22:51:11.602527 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:51:11.602496 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="297029e1-47c7-4aca-8699-4d94f72cd104" path="/var/lib/kubelet/pods/297029e1-47c7-4aca-8699-4d94f72cd104/volumes" Oct 30 22:51:27.910571 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:51:27.910528 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:51:57.909988 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:51:57.909904 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:52:00.155997 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:52:00.155963 6006 scope.go:122] "RemoveContainer" containerID="a9608efcb311cf1ff9637115e93938c91095fd965e1fdcb338d9d97819f358f9" Oct 30 22:52:27.909989 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:52:27.909946 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:52:41.278898 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:52:41.278856 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:52:57.909480 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:52:57.909439 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:52:58.665600 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:52:58.665559 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:53:05.986546 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:53:05.986434 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-6cmps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:53:05.986546 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:53:05.986503 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" podUID="2c3b94ab-018e-480f-b12c-faa5544fc2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:53:06.434630 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:53:06.434592 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:53:27.860072 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:53:27.860035 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:53:27.912297 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:53:27.912238 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:53:47.354168 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:53:47.354120 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:53:57.910890 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:53:57.910839 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:54:06.030535 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:06.030478 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:54:27.909721 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:54:27.909676 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:54:31.374446 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:31.374408 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:54:46.517328 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:46.517207 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f8c973b-x74kl"] Oct 30 22:54:46.517754 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:46.517675 6006 cpu_manager.go:414] "containerMap: removing container" podUID="297029e1-47c7-4aca-8699-4d94f72cd104" containerName="manager" Oct 30 22:54:46.517754 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:46.517687 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="297029e1-47c7-4aca-8699-4d94f72cd104" containerName="manager" Oct 30 22:54:46.517754 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:46.517705 6006 cpu_manager.go:414] "containerMap: removing container" podUID="2d8abf92-a63c-4368-aaf7-a2008135702b" containerName="revoke-keys" Oct 30 22:54:46.517754 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:46.517710 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d8abf92-a63c-4368-aaf7-a2008135702b" containerName="revoke-keys" Oct 30 22:54:46.517889 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:46.517768 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="297029e1-47c7-4aca-8699-4d94f72cd104" containerName="manager" Oct 30 22:54:46.517889 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:46.517778 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d8abf92-a63c-4368-aaf7-a2008135702b" containerName="revoke-keys" Oct 30 22:54:46.520826 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:46.520799 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8c973b-x74kl" Oct 30 22:54:46.570280 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:46.570210 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:54:46.598245 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:46.598199 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fbd750b-b1c2-478a-bfbc-3175d10e00e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f8c973b-x74kl\" (UID: \"5fbd750b-b1c2-478a-bfbc-3175d10e00e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8c973b-x74kl" Oct 30 22:54:46.598443 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:46.598281 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k99bt\" (UniqueName: \"kubernetes.io/projected/5fbd750b-b1c2-478a-bfbc-3175d10e00e9-kube-api-access-k99bt\") pod \"maas-api-revoke-sub-5f8c973b-x74kl\" (UID: \"5fbd750b-b1c2-478a-bfbc-3175d10e00e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8c973b-x74kl" Oct 30 22:54:46.598532 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:46.598502 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fbd750b-b1c2-478a-bfbc-3175d10e00e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f8c973b-x74kl\" (UID: \"5fbd750b-b1c2-478a-bfbc-3175d10e00e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8c973b-x74kl" Oct 30 22:54:46.616751 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:46.616715 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:54:46.616751 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:46.616722 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:54:46.616963 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:46.616910 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:54:46.618364 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:46.618343 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f8c973b-x74kl"] Oct 30 22:54:46.700066 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:46.700026 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fbd750b-b1c2-478a-bfbc-3175d10e00e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f8c973b-x74kl\" (UID: \"5fbd750b-b1c2-478a-bfbc-3175d10e00e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8c973b-x74kl" Oct 30 22:54:46.700331 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:46.700090 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fbd750b-b1c2-478a-bfbc-3175d10e00e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f8c973b-x74kl\" (UID: \"5fbd750b-b1c2-478a-bfbc-3175d10e00e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8c973b-x74kl" Oct 30 22:54:46.700331 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:46.700139 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k99bt\" (UniqueName: \"kubernetes.io/projected/5fbd750b-b1c2-478a-bfbc-3175d10e00e9-kube-api-access-k99bt\") pod \"maas-api-revoke-sub-5f8c973b-x74kl\" (UID: \"5fbd750b-b1c2-478a-bfbc-3175d10e00e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8c973b-x74kl" Oct 30 22:54:46.700845 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:46.700818 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fbd750b-b1c2-478a-bfbc-3175d10e00e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f8c973b-x74kl\" (UID: \"5fbd750b-b1c2-478a-bfbc-3175d10e00e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8c973b-x74kl" Oct 30 22:54:46.702597 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:46.702570 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fbd750b-b1c2-478a-bfbc-3175d10e00e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f8c973b-x74kl\" (UID: \"5fbd750b-b1c2-478a-bfbc-3175d10e00e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8c973b-x74kl" Oct 30 22:54:46.758700 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:46.758666 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k99bt\" (UniqueName: \"kubernetes.io/projected/5fbd750b-b1c2-478a-bfbc-3175d10e00e9-kube-api-access-k99bt\") pod \"maas-api-revoke-sub-5f8c973b-x74kl\" (UID: \"5fbd750b-b1c2-478a-bfbc-3175d10e00e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8c973b-x74kl" Oct 30 22:54:46.833904 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:46.833867 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8c973b-x74kl" Oct 30 22:54:47.204733 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:47.204520 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f8c973b-x74kl"] Oct 30 22:54:47.207239 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:54:47.207207 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fbd750b_b1c2_478a_bfbc_3175d10e00e9.slice/crio-26d1ca4c0d970ff42e4f3583428c642b72df5f7159cc17262ede00f927832d80 WatchSource:0}: Error finding container 26d1ca4c0d970ff42e4f3583428c642b72df5f7159cc17262ede00f927832d80: Status 404 returned error can't find the container with id 26d1ca4c0d970ff42e4f3583428c642b72df5f7159cc17262ede00f927832d80 Oct 30 22:54:48.214568 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:48.214531 6006 generic.go:356] "Generic (PLEG): container finished" podID="5fbd750b-b1c2-478a-bfbc-3175d10e00e9" containerID="3ddf290b06bc17ae4ca1b4e71adae0fb599dc7336bd09e2d515e3aa0bba604a0" exitCode=0 Oct 30 22:54:48.215015 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:48.214622 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8c973b-x74kl" event={"ID":"5fbd750b-b1c2-478a-bfbc-3175d10e00e9","Type":"ContainerDied","Data":"3ddf290b06bc17ae4ca1b4e71adae0fb599dc7336bd09e2d515e3aa0bba604a0"} Oct 30 22:54:48.215015 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:48.214726 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8c973b-x74kl" event={"ID":"5fbd750b-b1c2-478a-bfbc-3175d10e00e9","Type":"ContainerStarted","Data":"26d1ca4c0d970ff42e4f3583428c642b72df5f7159cc17262ede00f927832d80"} Oct 30 22:54:49.376688 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:49.376657 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8c973b-x74kl" Oct 30 22:54:49.431603 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:49.431544 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fbd750b-b1c2-478a-bfbc-3175d10e00e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fbd750b-b1c2-478a-bfbc-3175d10e00e9-maas-api-xks-ca\") pod \"5fbd750b-b1c2-478a-bfbc-3175d10e00e9\" (UID: \"5fbd750b-b1c2-478a-bfbc-3175d10e00e9\") " Oct 30 22:54:49.432689 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:49.432637 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fbd750b-b1c2-478a-bfbc-3175d10e00e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fbd750b-b1c2-478a-bfbc-3175d10e00e9-maas-api-openshift-ca\") pod \"5fbd750b-b1c2-478a-bfbc-3175d10e00e9\" (UID: \"5fbd750b-b1c2-478a-bfbc-3175d10e00e9\") " Oct 30 22:54:49.433102 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:49.433035 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fbd750b-b1c2-478a-bfbc-3175d10e00e9-kube-api-access-k99bt\" (UniqueName: \"kubernetes.io/projected/5fbd750b-b1c2-478a-bfbc-3175d10e00e9-kube-api-access-k99bt\") pod \"5fbd750b-b1c2-478a-bfbc-3175d10e00e9\" (UID: \"5fbd750b-b1c2-478a-bfbc-3175d10e00e9\") " Oct 30 22:54:49.434435 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:49.434389 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fbd750b-b1c2-478a-bfbc-3175d10e00e9-maas-api-openshift-ca" pod "5fbd750b-b1c2-478a-bfbc-3175d10e00e9" (UID: "5fbd750b-b1c2-478a-bfbc-3175d10e00e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:54:49.438423 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:49.438368 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fbd750b-b1c2-478a-bfbc-3175d10e00e9-maas-api-xks-ca" pod "5fbd750b-b1c2-478a-bfbc-3175d10e00e9" (UID: "5fbd750b-b1c2-478a-bfbc-3175d10e00e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:54:49.439733 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:49.439668 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fbd750b-b1c2-478a-bfbc-3175d10e00e9-kube-api-access-k99bt" pod "5fbd750b-b1c2-478a-bfbc-3175d10e00e9" (UID: "5fbd750b-b1c2-478a-bfbc-3175d10e00e9"). InnerVolumeSpecName "kube-api-access-k99bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:54:49.535686 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:49.535589 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fbd750b-b1c2-478a-bfbc-3175d10e00e9-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:49.535686 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:49.535636 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fbd750b-b1c2-478a-bfbc-3175d10e00e9-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:49.535686 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:49.535654 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k99bt\" (UniqueName: \"kubernetes.io/projected/5fbd750b-b1c2-478a-bfbc-3175d10e00e9-kube-api-access-k99bt\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:50.226302 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:50.226263 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8c973b-x74kl" Oct 30 22:54:50.226302 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:50.226288 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8c973b-x74kl" event={"ID":"5fbd750b-b1c2-478a-bfbc-3175d10e00e9","Type":"ContainerDied","Data":"26d1ca4c0d970ff42e4f3583428c642b72df5f7159cc17262ede00f927832d80"} Oct 30 22:54:50.226524 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:50.226364 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26d1ca4c0d970ff42e4f3583428c642b72df5f7159cc17262ede00f927832d80" Oct 30 22:54:57.910194 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:54:57.910155 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:54:58.205550 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:58.205459 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj"] Oct 30 22:54:58.205910 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:58.205897 6006 cpu_manager.go:414] "containerMap: removing container" podUID="5fbd750b-b1c2-478a-bfbc-3175d10e00e9" containerName="revoke-keys" Oct 30 22:54:58.205963 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:58.205911 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fbd750b-b1c2-478a-bfbc-3175d10e00e9" containerName="revoke-keys" Oct 30 22:54:58.206022 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:58.206013 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fbd750b-b1c2-478a-bfbc-3175d10e00e9" containerName="revoke-keys" Oct 30 22:54:58.209546 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:58.209524 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj" Oct 30 22:54:58.250416 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:58.250387 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 22:54:58.260397 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:58.260362 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:54:58.282658 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:58.282631 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 22:54:58.305598 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:58.305562 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:54:58.305598 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:58.305570 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:54:58.305857 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:58.305694 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5rvcj\"" Oct 30 22:54:58.307470 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:58.307451 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj"] Oct 30 22:54:58.325086 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:58.325046 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p8vl\" (UniqueName: \"kubernetes.io/projected/0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3-kube-api-access-4p8vl\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj\" (UID: \"0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj" Oct 30 22:54:58.325317 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:58.325092 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj\" (UID: \"0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj" Oct 30 22:54:58.325317 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:58.325224 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj\" (UID: \"0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj" Oct 30 22:54:58.426755 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:58.426712 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p8vl\" (UniqueName: \"kubernetes.io/projected/0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3-kube-api-access-4p8vl\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj\" (UID: \"0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj" Oct 30 22:54:58.426755 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:58.426762 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj\" (UID: \"0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj" Oct 30 22:54:58.427020 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:58.426811 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj\" (UID: \"0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj" Oct 30 22:54:58.427580 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:58.427557 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj\" (UID: \"0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj" Oct 30 22:54:58.429483 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:58.429460 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj\" (UID: \"0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj" Oct 30 22:54:58.485373 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:58.485302 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p8vl\" (UniqueName: \"kubernetes.io/projected/0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3-kube-api-access-4p8vl\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj\" (UID: \"0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj" Oct 30 22:54:58.522598 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:58.522563 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj" Oct 30 22:54:58.889991 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:58.889954 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj"] Oct 30 22:54:58.894411 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:54:58.894377 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3e20b1_e7bc_4356_97f3_aa3a9d2762a3.slice/crio-ee31399f0265b8c28dac7de401f7e4ee2a971aebc8fa338581f2fdc7286b2019 WatchSource:0}: Error finding container ee31399f0265b8c28dac7de401f7e4ee2a971aebc8fa338581f2fdc7286b2019: Status 404 returned error can't find the container with id ee31399f0265b8c28dac7de401f7e4ee2a971aebc8fa338581f2fdc7286b2019 Oct 30 22:54:59.271197 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:59.271159 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj" event={"ID":"0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3","Type":"ContainerStarted","Data":"158ac2123589baa66e2620ac52aa6601658085ddcd9f653814986ff6d136dd57"} Oct 30 22:54:59.271633 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:59.271225 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj" event={"ID":"0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3","Type":"ContainerStarted","Data":"ee31399f0265b8c28dac7de401f7e4ee2a971aebc8fa338581f2fdc7286b2019"} Oct 30 22:54:59.389650 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:59.389583 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj" podStartSLOduration=1.389564385 podStartE2EDuration="1.389564385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:54:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:54:59.389007814 +0000 UTC m=+3901.941184115" watchObservedRunningTime="2026-10-30 22:54:59.389564385 +0000 UTC m=+3901.941740682" Oct 30 22:54:59.554987 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:54:59.554947 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:55:00.277863 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:55:00.277771 6006 generic.go:356] "Generic (PLEG): container finished" podID="0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3" containerID="158ac2123589baa66e2620ac52aa6601658085ddcd9f653814986ff6d136dd57" exitCode=0 Oct 30 22:55:00.277863 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:55:00.277851 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj" event={"ID":"0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3","Type":"ContainerDied","Data":"158ac2123589baa66e2620ac52aa6601658085ddcd9f653814986ff6d136dd57"} Oct 30 22:55:01.418265 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:55:01.418222 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj" Oct 30 22:55:01.467570 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:55:01.467532 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3-kube-api-access-4p8vl\" (UniqueName: \"kubernetes.io/projected/0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3-kube-api-access-4p8vl\") pod \"0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3\" (UID: \"0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3\") " Oct 30 22:55:01.467771 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:55:01.467630 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3-maas-api-xks-ca\") pod \"0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3\" (UID: \"0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3\") " Oct 30 22:55:01.467771 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:55:01.467715 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3-maas-api-openshift-ca\") pod \"0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3\" (UID: \"0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3\") " Oct 30 22:55:01.468141 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:55:01.468114 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3-maas-api-openshift-ca" pod "0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3" (UID: "0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:55:01.469758 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:55:01.469729 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3-maas-api-xks-ca" pod "0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3" (UID: "0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:55:01.469870 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:55:01.469807 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3-kube-api-access-4p8vl" pod "0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3" (UID: "0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3"). InnerVolumeSpecName "kube-api-access-4p8vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:55:01.569493 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:55:01.569461 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p8vl\" (UniqueName: \"kubernetes.io/projected/0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3-kube-api-access-4p8vl\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:01.569493 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:55:01.569490 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:01.569721 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:55:01.569505 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:02.289173 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:55:02.289138 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj" Oct 30 22:55:02.289407 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:55:02.289170 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj" event={"ID":"0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3","Type":"ContainerDied","Data":"ee31399f0265b8c28dac7de401f7e4ee2a971aebc8fa338581f2fdc7286b2019"} Oct 30 22:55:02.289407 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:55:02.289286 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee31399f0265b8c28dac7de401f7e4ee2a971aebc8fa338581f2fdc7286b2019" Oct 30 22:55:13.521117 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:55:13.521070 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj"] Oct 30 22:55:13.566664 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:55:13.566632 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4334bf-9w7hj"] Oct 30 22:55:13.567937 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:55:13.567914 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:55:13.601858 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:55:13.601811 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3" path="/var/lib/kubelet/pods/0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3/volumes" Oct 30 22:55:27.910148 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:55:27.910107 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:55:32.742634 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:55:32.742593 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:55:35.987481 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:55:35.987428 6006 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-6cmps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:55:35.987912 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:55:35.987514 6006 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6cmps" podUID="2c3b94ab-018e-480f-b12c-faa5544fc2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:55:57.910403 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:55:57.910363 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:56:14.671142 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:56:14.671059 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:56:27.909636 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:56:27.909592 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:56:44.970771 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:56:44.970728 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:56:57.909404 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:56:57.909361 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:56:59.663972 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:56:59.663930 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:57:13.488284 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:57:13.488226 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:57:25.870707 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:57:25.870663 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:57:27.981221 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:57:27.981180 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:57:37.057822 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:57:37.057738 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:57:57.910399 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:57:57.910358 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:57:58.267714 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:57:58.267626 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:58:22.975418 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:22.975377 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:58:27.910346 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:58:27.910301 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:58:46.328198 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:46.328161 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l"] Oct 30 22:58:46.328621 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:46.328570 6006 cpu_manager.go:414] "containerMap: removing container" podUID="0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3" containerName="revoke-keys" Oct 30 22:58:46.328621 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:46.328582 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3" containerName="revoke-keys" Oct 30 22:58:46.328701 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:46.328680 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d3e20b1-e7bc-4356-97f3-aa3a9d2762a3" containerName="revoke-keys" Oct 30 22:58:46.331798 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:46.331776 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l" Oct 30 22:58:46.381077 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:46.381035 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:58:46.427746 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:46.427707 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:58:46.427926 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:46.427770 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k2h55\"" Oct 30 22:58:46.427926 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:46.427786 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:58:46.429587 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:46.429564 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l"] Oct 30 22:58:46.433422 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:46.433398 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b358c750-a61d-405a-890c-0f132f253b0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l\" (UID: \"b358c750-a61d-405a-890c-0f132f253b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l" Oct 30 22:58:46.433482 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:46.433445 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b358c750-a61d-405a-890c-0f132f253b0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l\" (UID: \"b358c750-a61d-405a-890c-0f132f253b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l" Oct 30 22:58:46.433614 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:46.433593 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6kt7\" (UniqueName: \"kubernetes.io/projected/b358c750-a61d-405a-890c-0f132f253b0c-kube-api-access-b6kt7\") pod \"maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l\" (UID: \"b358c750-a61d-405a-890c-0f132f253b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l" Oct 30 22:58:46.534401 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:46.534363 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b358c750-a61d-405a-890c-0f132f253b0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l\" (UID: \"b358c750-a61d-405a-890c-0f132f253b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l" Oct 30 22:58:46.534632 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:46.534552 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6kt7\" (UniqueName: \"kubernetes.io/projected/b358c750-a61d-405a-890c-0f132f253b0c-kube-api-access-b6kt7\") pod \"maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l\" (UID: \"b358c750-a61d-405a-890c-0f132f253b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l" Oct 30 22:58:46.534710 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:46.534638 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b358c750-a61d-405a-890c-0f132f253b0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l\" (UID: \"b358c750-a61d-405a-890c-0f132f253b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l" Oct 30 22:58:46.535146 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:46.535119 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b358c750-a61d-405a-890c-0f132f253b0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l\" (UID: \"b358c750-a61d-405a-890c-0f132f253b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l" Oct 30 22:58:46.537070 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:46.537053 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b358c750-a61d-405a-890c-0f132f253b0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l\" (UID: \"b358c750-a61d-405a-890c-0f132f253b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l" Oct 30 22:58:46.593635 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:46.593555 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6kt7\" (UniqueName: \"kubernetes.io/projected/b358c750-a61d-405a-890c-0f132f253b0c-kube-api-access-b6kt7\") pod \"maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l\" (UID: \"b358c750-a61d-405a-890c-0f132f253b0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l" Oct 30 22:58:46.644428 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:46.644381 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l" Oct 30 22:58:47.014900 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:47.014874 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l"] Oct 30 22:58:47.020915 ip-10-0-135-7 kubenswrapper[6006]: W1030 22:58:47.020545 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb358c750_a61d_405a_890c_0f132f253b0c.slice/crio-b3b4a051e75c45440ef8a1c4f35e97e141352c4d154021b6a870b4022350060e WatchSource:0}: Error finding container b3b4a051e75c45440ef8a1c4f35e97e141352c4d154021b6a870b4022350060e: Status 404 returned error can't find the container with id b3b4a051e75c45440ef8a1c4f35e97e141352c4d154021b6a870b4022350060e Oct 30 22:58:47.589417 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:47.589323 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l" event={"ID":"b358c750-a61d-405a-890c-0f132f253b0c","Type":"ContainerStarted","Data":"a5b4b0cb81079c931e6a507b88bf0bf2a5ef6fe17e88fc1917032dc4c4339fcf"} Oct 30 22:58:47.589417 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:47.589375 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l" event={"ID":"b358c750-a61d-405a-890c-0f132f253b0c","Type":"ContainerStarted","Data":"b3b4a051e75c45440ef8a1c4f35e97e141352c4d154021b6a870b4022350060e"} Oct 30 22:58:47.702570 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:47.702503 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l" podStartSLOduration=1.702484422 podStartE2EDuration="1.702484422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:58:47.701885899 +0000 UTC m=+4130.254062199" watchObservedRunningTime="2026-10-30 22:58:47.702484422 +0000 UTC m=+4130.254660723" Oct 30 22:58:48.596142 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:48.596098 6006 generic.go:356] "Generic (PLEG): container finished" podID="b358c750-a61d-405a-890c-0f132f253b0c" containerID="a5b4b0cb81079c931e6a507b88bf0bf2a5ef6fe17e88fc1917032dc4c4339fcf" exitCode=0 Oct 30 22:58:48.596612 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:48.596192 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l" event={"ID":"b358c750-a61d-405a-890c-0f132f253b0c","Type":"ContainerDied","Data":"a5b4b0cb81079c931e6a507b88bf0bf2a5ef6fe17e88fc1917032dc4c4339fcf"} Oct 30 22:58:49.745054 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:49.745024 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l" Oct 30 22:58:49.869132 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:49.868963 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b358c750-a61d-405a-890c-0f132f253b0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b358c750-a61d-405a-890c-0f132f253b0c-maas-api-openshift-ca\") pod \"b358c750-a61d-405a-890c-0f132f253b0c\" (UID: \"b358c750-a61d-405a-890c-0f132f253b0c\") " Oct 30 22:58:49.869132 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:49.869061 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b358c750-a61d-405a-890c-0f132f253b0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b358c750-a61d-405a-890c-0f132f253b0c-maas-api-xks-ca\") pod \"b358c750-a61d-405a-890c-0f132f253b0c\" (UID: \"b358c750-a61d-405a-890c-0f132f253b0c\") " Oct 30 22:58:49.869341 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:49.869192 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b358c750-a61d-405a-890c-0f132f253b0c-kube-api-access-b6kt7\" (UniqueName: \"kubernetes.io/projected/b358c750-a61d-405a-890c-0f132f253b0c-kube-api-access-b6kt7\") pod \"b358c750-a61d-405a-890c-0f132f253b0c\" (UID: \"b358c750-a61d-405a-890c-0f132f253b0c\") " Oct 30 22:58:49.869580 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:49.869539 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b358c750-a61d-405a-890c-0f132f253b0c-maas-api-openshift-ca" pod "b358c750-a61d-405a-890c-0f132f253b0c" (UID: "b358c750-a61d-405a-890c-0f132f253b0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:58:49.871875 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:49.871840 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b358c750-a61d-405a-890c-0f132f253b0c-kube-api-access-b6kt7" pod "b358c750-a61d-405a-890c-0f132f253b0c" (UID: "b358c750-a61d-405a-890c-0f132f253b0c"). InnerVolumeSpecName "kube-api-access-b6kt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:58:49.872633 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:49.872604 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b358c750-a61d-405a-890c-0f132f253b0c-maas-api-xks-ca" pod "b358c750-a61d-405a-890c-0f132f253b0c" (UID: "b358c750-a61d-405a-890c-0f132f253b0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:58:49.970503 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:49.970466 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6kt7\" (UniqueName: \"kubernetes.io/projected/b358c750-a61d-405a-890c-0f132f253b0c-kube-api-access-b6kt7\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:58:49.970503 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:49.970498 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b358c750-a61d-405a-890c-0f132f253b0c-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:58:49.970503 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:49.970511 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b358c750-a61d-405a-890c-0f132f253b0c-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:58:50.608881 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:50.608838 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l" event={"ID":"b358c750-a61d-405a-890c-0f132f253b0c","Type":"ContainerDied","Data":"b3b4a051e75c45440ef8a1c4f35e97e141352c4d154021b6a870b4022350060e"} Oct 30 22:58:50.608881 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:50.608861 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l" Oct 30 22:58:50.608881 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:58:50.608888 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3b4a051e75c45440ef8a1c4f35e97e141352c4d154021b6a870b4022350060e" Oct 30 22:58:57.910029 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:58:57.909987 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:59:07.628873 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:59:07.628786 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l"] Oct 30 22:59:07.675444 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:59:07.675409 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f23d4bb3-4ft4l"] Oct 30 22:59:09.602958 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:59:09.602915 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b358c750-a61d-405a-890c-0f132f253b0c" path="/var/lib/kubelet/pods/b358c750-a61d-405a-890c-0f132f253b0c/volumes" Oct 30 22:59:09.983540 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:59:09.983446 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:59:27.910783 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:59:27.910736 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:59:33.167153 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:59:33.167104 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 22:59:50.046692 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:59:50.046653 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f8c973b-x74kl"] Oct 30 22:59:50.093260 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:59:50.093220 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f8c973b-x74kl"] Oct 30 22:59:51.602515 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:59:51.602478 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fbd750b-b1c2-478a-bfbc-3175d10e00e9" path="/var/lib/kubelet/pods/5fbd750b-b1c2-478a-bfbc-3175d10e00e9/volumes" Oct 30 22:59:57.911817 ip-10-0-135-7 kubenswrapper[6006]: E1030 22:59:57.911773 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 22:59:58.280880 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:59:58.280806 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 22:59:58.321827 ip-10-0-135-7 kubenswrapper[6006]: I1030 22:59:58.321797 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 23:00:00.179851 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:00.179818 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9"] Oct 30 23:00:00.180277 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:00.180265 6006 cpu_manager.go:414] "containerMap: removing container" podUID="b358c750-a61d-405a-890c-0f132f253b0c" containerName="revoke-keys" Oct 30 23:00:00.180333 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:00.180278 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b358c750-a61d-405a-890c-0f132f253b0c" containerName="revoke-keys" Oct 30 23:00:00.180379 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:00.180354 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="b358c750-a61d-405a-890c-0f132f253b0c" containerName="revoke-keys" Oct 30 23:00:00.184623 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:00.184600 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9" Oct 30 23:00:00.198030 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:00.198003 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pwxp\" (UniqueName: \"kubernetes.io/projected/170acbb8-9b1e-451f-a812-4390a7cc6a06-kube-api-access-8pwxp\") pod \"maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9\" (UID: \"170acbb8-9b1e-451f-a812-4390a7cc6a06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9" Oct 30 23:00:00.198187 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:00.198070 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/170acbb8-9b1e-451f-a812-4390a7cc6a06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9\" (UID: \"170acbb8-9b1e-451f-a812-4390a7cc6a06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9" Oct 30 23:00:00.284742 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:00.284702 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 23:00:00.285000 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:00.284731 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k2h55\"" Oct 30 23:00:00.285000 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:00.284851 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 23:00:00.286713 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:00.286682 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9"] Oct 30 23:00:00.298767 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:00.298728 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pwxp\" (UniqueName: \"kubernetes.io/projected/170acbb8-9b1e-451f-a812-4390a7cc6a06-kube-api-access-8pwxp\") pod \"maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9\" (UID: \"170acbb8-9b1e-451f-a812-4390a7cc6a06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9" Oct 30 23:00:00.298947 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:00.298788 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/170acbb8-9b1e-451f-a812-4390a7cc6a06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9\" (UID: \"170acbb8-9b1e-451f-a812-4390a7cc6a06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9" Oct 30 23:00:00.299876 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:00.299846 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/170acbb8-9b1e-451f-a812-4390a7cc6a06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9\" (UID: \"170acbb8-9b1e-451f-a812-4390a7cc6a06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9" Oct 30 23:00:00.357969 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:00.357933 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pwxp\" (UniqueName: \"kubernetes.io/projected/170acbb8-9b1e-451f-a812-4390a7cc6a06-kube-api-access-8pwxp\") pod \"maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9\" (UID: \"170acbb8-9b1e-451f-a812-4390a7cc6a06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9" Oct 30 23:00:00.499009 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:00.498920 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9" Oct 30 23:00:00.865169 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:00.865142 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9"] Oct 30 23:00:00.868167 ip-10-0-135-7 kubenswrapper[6006]: W1030 23:00:00.868123 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod170acbb8_9b1e_451f_a812_4390a7cc6a06.slice/crio-0306ac43d4206bf04375206e0b90445306eb96bf4a5969a39e0323728c12014b WatchSource:0}: Error finding container 0306ac43d4206bf04375206e0b90445306eb96bf4a5969a39e0323728c12014b: Status 404 returned error can't find the container with id 0306ac43d4206bf04375206e0b90445306eb96bf4a5969a39e0323728c12014b Oct 30 23:00:00.980962 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:00.980923 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9" event={"ID":"170acbb8-9b1e-451f-a812-4390a7cc6a06","Type":"ContainerStarted","Data":"0306ac43d4206bf04375206e0b90445306eb96bf4a5969a39e0323728c12014b"} Oct 30 23:00:01.987954 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:01.987859 6006 generic.go:356] "Generic (PLEG): container finished" podID="170acbb8-9b1e-451f-a812-4390a7cc6a06" containerID="3ff22c48214460473e3397c79d466f60141075e9968ad554ff1c23e758202c41" exitCode=60 Oct 30 23:00:01.988342 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:01.987951 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9" event={"ID":"170acbb8-9b1e-451f-a812-4390a7cc6a06","Type":"ContainerDied","Data":"3ff22c48214460473e3397c79d466f60141075e9968ad554ff1c23e758202c41"} Oct 30 23:00:01.988342 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:01.988229 6006 scope.go:122] "RemoveContainer" containerID="3ff22c48214460473e3397c79d466f60141075e9968ad554ff1c23e758202c41" Oct 30 23:00:02.995449 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:02.995413 6006 generic.go:356] "Generic (PLEG): container finished" podID="170acbb8-9b1e-451f-a812-4390a7cc6a06" containerID="dfcaa64736bd0682e527681277749ff261347f32ce244dc202b2a61692535366" exitCode=60 Oct 30 23:00:02.995925 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:02.995498 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9" event={"ID":"170acbb8-9b1e-451f-a812-4390a7cc6a06","Type":"ContainerDied","Data":"dfcaa64736bd0682e527681277749ff261347f32ce244dc202b2a61692535366"} Oct 30 23:00:02.995925 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:02.995565 6006 scope.go:122] "RemoveContainer" containerID="3ff22c48214460473e3397c79d466f60141075e9968ad554ff1c23e758202c41" Oct 30 23:00:02.995925 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:02.995824 6006 scope.go:122] "RemoveContainer" containerID="dfcaa64736bd0682e527681277749ff261347f32ce244dc202b2a61692535366" Oct 30 23:00:02.996150 ip-10-0-135-7 kubenswrapper[6006]: E1030 23:00:02.996129 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9_odh-ai-gateway-infra(170acbb8-9b1e-451f-a812-4390a7cc6a06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9" podUID="170acbb8-9b1e-451f-a812-4390a7cc6a06" Oct 30 23:00:04.002662 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:04.002634 6006 scope.go:122] "RemoveContainer" containerID="dfcaa64736bd0682e527681277749ff261347f32ce244dc202b2a61692535366" Oct 30 23:00:04.003058 ip-10-0-135-7 kubenswrapper[6006]: E1030 23:00:04.002879 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9_odh-ai-gateway-infra(170acbb8-9b1e-451f-a812-4390a7cc6a06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9" podUID="170acbb8-9b1e-451f-a812-4390a7cc6a06" Oct 30 23:00:15.596777 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:15.596736 6006 scope.go:122] "RemoveContainer" containerID="dfcaa64736bd0682e527681277749ff261347f32ce244dc202b2a61692535366" Oct 30 23:00:16.065922 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:16.065883 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9" event={"ID":"170acbb8-9b1e-451f-a812-4390a7cc6a06","Type":"ContainerStarted","Data":"29f0b7b8c068fc4a8a9f844e6bbadef14775e790182e9818128af4578d64c959"} Oct 30 23:00:16.179017 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:16.178910 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9" podStartSLOduration=16.178895226 podStartE2EDuration="16.178895226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:16.17801559 +0000 UTC m=+4218.730191891" watchObservedRunningTime="2026-10-30 23:00:16.178895226 +0000 UTC m=+4218.731071539" Oct 30 23:00:17.073053 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:17.073016 6006 generic.go:356] "Generic (PLEG): container finished" podID="170acbb8-9b1e-451f-a812-4390a7cc6a06" containerID="29f0b7b8c068fc4a8a9f844e6bbadef14775e790182e9818128af4578d64c959" exitCode=60 Oct 30 23:00:17.073558 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:17.073071 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9" event={"ID":"170acbb8-9b1e-451f-a812-4390a7cc6a06","Type":"ContainerDied","Data":"29f0b7b8c068fc4a8a9f844e6bbadef14775e790182e9818128af4578d64c959"} Oct 30 23:00:17.073558 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:17.073118 6006 scope.go:122] "RemoveContainer" containerID="dfcaa64736bd0682e527681277749ff261347f32ce244dc202b2a61692535366" Oct 30 23:00:17.073558 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:17.073485 6006 scope.go:122] "RemoveContainer" containerID="29f0b7b8c068fc4a8a9f844e6bbadef14775e790182e9818128af4578d64c959" Oct 30 23:00:17.073804 ip-10-0-135-7 kubenswrapper[6006]: E1030 23:00:17.073783 6006 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9_odh-ai-gateway-infra(170acbb8-9b1e-451f-a812-4390a7cc6a06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9" podUID="170acbb8-9b1e-451f-a812-4390a7cc6a06" Oct 30 23:00:17.197891 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:17.197848 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9"] Oct 30 23:00:18.234337 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:18.234306 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9" Oct 30 23:00:18.373886 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:18.373796 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/170acbb8-9b1e-451f-a812-4390a7cc6a06-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/170acbb8-9b1e-451f-a812-4390a7cc6a06-maas-api-service-ca\") pod \"170acbb8-9b1e-451f-a812-4390a7cc6a06\" (UID: \"170acbb8-9b1e-451f-a812-4390a7cc6a06\") " Oct 30 23:00:18.373886 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:18.373847 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/170acbb8-9b1e-451f-a812-4390a7cc6a06-kube-api-access-8pwxp\" (UniqueName: \"kubernetes.io/projected/170acbb8-9b1e-451f-a812-4390a7cc6a06-kube-api-access-8pwxp\") pod \"170acbb8-9b1e-451f-a812-4390a7cc6a06\" (UID: \"170acbb8-9b1e-451f-a812-4390a7cc6a06\") " Oct 30 23:00:18.374318 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:18.374295 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/170acbb8-9b1e-451f-a812-4390a7cc6a06-maas-api-service-ca" pod "170acbb8-9b1e-451f-a812-4390a7cc6a06" (UID: "170acbb8-9b1e-451f-a812-4390a7cc6a06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:00:18.376193 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:18.376162 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/170acbb8-9b1e-451f-a812-4390a7cc6a06-kube-api-access-8pwxp" pod "170acbb8-9b1e-451f-a812-4390a7cc6a06" (UID: "170acbb8-9b1e-451f-a812-4390a7cc6a06"). InnerVolumeSpecName "kube-api-access-8pwxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:00:18.475231 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:18.475189 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/170acbb8-9b1e-451f-a812-4390a7cc6a06-maas-api-service-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:00:18.475231 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:18.475221 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pwxp\" (UniqueName: \"kubernetes.io/projected/170acbb8-9b1e-451f-a812-4390a7cc6a06-kube-api-access-8pwxp\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:00:19.085789 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:19.085758 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9" Oct 30 23:00:19.085789 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:19.085777 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9" event={"ID":"170acbb8-9b1e-451f-a812-4390a7cc6a06","Type":"ContainerDied","Data":"0306ac43d4206bf04375206e0b90445306eb96bf4a5969a39e0323728c12014b"} Oct 30 23:00:19.085996 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:19.085834 6006 scope.go:122] "RemoveContainer" containerID="29f0b7b8c068fc4a8a9f844e6bbadef14775e790182e9818128af4578d64c959" Oct 30 23:00:19.241269 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:19.241209 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9"] Oct 30 23:00:19.287714 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:19.287666 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-31179c-29890020-nt2x9"] Oct 30 23:00:19.601978 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:00:19.601940 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="170acbb8-9b1e-451f-a812-4390a7cc6a06" path="/var/lib/kubelet/pods/170acbb8-9b1e-451f-a812-4390a7cc6a06/volumes" Oct 30 23:00:27.909548 ip-10-0-135-7 kubenswrapper[6006]: E1030 23:00:27.909506 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 23:00:57.910692 ip-10-0-135-7 kubenswrapper[6006]: E1030 23:00:57.910607 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 23:01:00.244785 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:00.244753 6006 scope.go:122] "RemoveContainer" containerID="158ac2123589baa66e2620ac52aa6601658085ddcd9f653814986ff6d136dd57" Oct 30 23:01:00.253547 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:00.253522 6006 scope.go:122] "RemoveContainer" containerID="3ddf290b06bc17ae4ca1b4e71adae0fb599dc7336bd09e2d515e3aa0bba604a0" Oct 30 23:01:04.248953 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:04.248914 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292"] Oct 30 23:01:04.249391 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:04.249372 6006 cpu_manager.go:414] "containerMap: removing container" podUID="170acbb8-9b1e-451f-a812-4390a7cc6a06" containerName="cleanup" Oct 30 23:01:04.249391 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:04.249386 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="170acbb8-9b1e-451f-a812-4390a7cc6a06" containerName="cleanup" Oct 30 23:01:04.249470 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:04.249401 6006 cpu_manager.go:414] "containerMap: removing container" podUID="170acbb8-9b1e-451f-a812-4390a7cc6a06" containerName="cleanup" Oct 30 23:01:04.249470 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:04.249407 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="170acbb8-9b1e-451f-a812-4390a7cc6a06" containerName="cleanup" Oct 30 23:01:04.249548 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:04.249497 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="170acbb8-9b1e-451f-a812-4390a7cc6a06" containerName="cleanup" Oct 30 23:01:04.249548 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:04.249512 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="170acbb8-9b1e-451f-a812-4390a7cc6a06" containerName="cleanup" Oct 30 23:01:04.252622 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:04.252601 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292" Oct 30 23:01:04.333064 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:04.333023 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b620ecf-2617-455e-af1f-27286b844486-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292\" (UID: \"9b620ecf-2617-455e-af1f-27286b844486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292" Oct 30 23:01:04.333264 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:04.333069 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b620ecf-2617-455e-af1f-27286b844486-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292\" (UID: \"9b620ecf-2617-455e-af1f-27286b844486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292" Oct 30 23:01:04.333264 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:04.333155 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztgwj\" (UniqueName: \"kubernetes.io/projected/9b620ecf-2617-455e-af1f-27286b844486-kube-api-access-ztgwj\") pod \"maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292\" (UID: \"9b620ecf-2617-455e-af1f-27286b844486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292" Oct 30 23:01:04.348450 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:04.348417 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 23:01:04.348579 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:04.348498 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:01:04.348579 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:04.348510 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k2h55\"" Oct 30 23:01:04.348579 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:04.348525 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 23:01:04.350017 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:04.349995 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292"] Oct 30 23:01:04.434381 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:04.434349 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztgwj\" (UniqueName: \"kubernetes.io/projected/9b620ecf-2617-455e-af1f-27286b844486-kube-api-access-ztgwj\") pod \"maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292\" (UID: \"9b620ecf-2617-455e-af1f-27286b844486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292" Oct 30 23:01:04.434586 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:04.434497 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b620ecf-2617-455e-af1f-27286b844486-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292\" (UID: \"9b620ecf-2617-455e-af1f-27286b844486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292" Oct 30 23:01:04.434586 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:04.434531 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b620ecf-2617-455e-af1f-27286b844486-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292\" (UID: \"9b620ecf-2617-455e-af1f-27286b844486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292" Oct 30 23:01:04.435176 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:04.435150 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b620ecf-2617-455e-af1f-27286b844486-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292\" (UID: \"9b620ecf-2617-455e-af1f-27286b844486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292" Oct 30 23:01:04.437118 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:04.437092 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b620ecf-2617-455e-af1f-27286b844486-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292\" (UID: \"9b620ecf-2617-455e-af1f-27286b844486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292" Oct 30 23:01:04.493479 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:04.493447 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztgwj\" (UniqueName: \"kubernetes.io/projected/9b620ecf-2617-455e-af1f-27286b844486-kube-api-access-ztgwj\") pod \"maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292\" (UID: \"9b620ecf-2617-455e-af1f-27286b844486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292" Oct 30 23:01:04.565436 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:04.565402 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292" Oct 30 23:01:04.935517 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:04.935324 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292"] Oct 30 23:01:04.939169 ip-10-0-135-7 kubenswrapper[6006]: W1030 23:01:04.939133 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b620ecf_2617_455e_af1f_27286b844486.slice/crio-07cdb538f631c92da7dbd50b47bad38d1304c14fc6c609ef4ec9964c4d529715 WatchSource:0}: Error finding container 07cdb538f631c92da7dbd50b47bad38d1304c14fc6c609ef4ec9964c4d529715: Status 404 returned error can't find the container with id 07cdb538f631c92da7dbd50b47bad38d1304c14fc6c609ef4ec9964c4d529715 Oct 30 23:01:05.320239 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:05.320200 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292" event={"ID":"9b620ecf-2617-455e-af1f-27286b844486","Type":"ContainerStarted","Data":"5d34f906245cddf86e6e018a4c1bd3f4f1138b02ae5925c2609416b9450ec2d4"} Oct 30 23:01:05.320703 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:05.320284 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292" event={"ID":"9b620ecf-2617-455e-af1f-27286b844486","Type":"ContainerStarted","Data":"07cdb538f631c92da7dbd50b47bad38d1304c14fc6c609ef4ec9964c4d529715"} Oct 30 23:01:05.430743 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:05.430682 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292" podStartSLOduration=1.430667629 podStartE2EDuration="1.430667629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:01:05.429654681 +0000 UTC m=+4267.981830981" watchObservedRunningTime="2026-10-30 23:01:05.430667629 +0000 UTC m=+4267.982843931" Oct 30 23:01:06.326305 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:06.326239 6006 generic.go:356] "Generic (PLEG): container finished" podID="9b620ecf-2617-455e-af1f-27286b844486" containerID="5d34f906245cddf86e6e018a4c1bd3f4f1138b02ae5925c2609416b9450ec2d4" exitCode=0 Oct 30 23:01:06.326809 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:06.326328 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292" event={"ID":"9b620ecf-2617-455e-af1f-27286b844486","Type":"ContainerDied","Data":"5d34f906245cddf86e6e018a4c1bd3f4f1138b02ae5925c2609416b9450ec2d4"} Oct 30 23:01:07.471100 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:07.471071 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292" Oct 30 23:01:07.568731 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:07.568694 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b620ecf-2617-455e-af1f-27286b844486-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b620ecf-2617-455e-af1f-27286b844486-maas-api-xks-ca\") pod \"9b620ecf-2617-455e-af1f-27286b844486\" (UID: \"9b620ecf-2617-455e-af1f-27286b844486\") " Oct 30 23:01:07.568917 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:07.568790 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b620ecf-2617-455e-af1f-27286b844486-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b620ecf-2617-455e-af1f-27286b844486-maas-api-openshift-ca\") pod \"9b620ecf-2617-455e-af1f-27286b844486\" (UID: \"9b620ecf-2617-455e-af1f-27286b844486\") " Oct 30 23:01:07.568917 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:07.568895 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b620ecf-2617-455e-af1f-27286b844486-kube-api-access-ztgwj\" (UniqueName: \"kubernetes.io/projected/9b620ecf-2617-455e-af1f-27286b844486-kube-api-access-ztgwj\") pod \"9b620ecf-2617-455e-af1f-27286b844486\" (UID: \"9b620ecf-2617-455e-af1f-27286b844486\") " Oct 30 23:01:07.569130 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:07.569109 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b620ecf-2617-455e-af1f-27286b844486-maas-api-openshift-ca" pod "9b620ecf-2617-455e-af1f-27286b844486" (UID: "9b620ecf-2617-455e-af1f-27286b844486"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:01:07.569378 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:07.569358 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b620ecf-2617-455e-af1f-27286b844486-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:01:07.570968 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:07.570945 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b620ecf-2617-455e-af1f-27286b844486-maas-api-xks-ca" pod "9b620ecf-2617-455e-af1f-27286b844486" (UID: "9b620ecf-2617-455e-af1f-27286b844486"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:01:07.571122 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:07.571106 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b620ecf-2617-455e-af1f-27286b844486-kube-api-access-ztgwj" pod "9b620ecf-2617-455e-af1f-27286b844486" (UID: "9b620ecf-2617-455e-af1f-27286b844486"). InnerVolumeSpecName "kube-api-access-ztgwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:01:07.670505 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:07.670467 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztgwj\" (UniqueName: \"kubernetes.io/projected/9b620ecf-2617-455e-af1f-27286b844486-kube-api-access-ztgwj\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:01:07.670505 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:07.670500 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b620ecf-2617-455e-af1f-27286b844486-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:01:08.337029 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:08.336999 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292" Oct 30 23:01:08.337217 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:08.336995 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292" event={"ID":"9b620ecf-2617-455e-af1f-27286b844486","Type":"ContainerDied","Data":"07cdb538f631c92da7dbd50b47bad38d1304c14fc6c609ef4ec9964c4d529715"} Oct 30 23:01:08.337217 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:08.337132 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07cdb538f631c92da7dbd50b47bad38d1304c14fc6c609ef4ec9964c4d529715" Oct 30 23:01:27.909984 ip-10-0-135-7 kubenswrapper[6006]: E1030 23:01:27.909946 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 23:01:31.278116 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:31.278074 6006 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292"] Oct 30 23:01:31.324643 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:31.324607 6006 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-31179c-nn292"] Oct 30 23:01:31.601990 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:31.601956 6006 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b620ecf-2617-455e-af1f-27286b844486" path="/var/lib/kubelet/pods/9b620ecf-2617-455e-af1f-27286b844486/volumes" Oct 30 23:01:34.194642 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:34.194605 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 23:01:45.665424 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:45.665382 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 23:01:57.444422 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:01:57.444377 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 23:01:57.916097 ip-10-0-135-7 kubenswrapper[6006]: E1030 23:01:57.916011 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 23:02:05.593394 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:05.593350 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-55fz5"] Oct 30 23:02:05.594305 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:05.594280 6006 cpu_manager.go:414] "containerMap: removing container" podUID="170acbb8-9b1e-451f-a812-4390a7cc6a06" containerName="cleanup" Oct 30 23:02:05.594305 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:05.594305 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="170acbb8-9b1e-451f-a812-4390a7cc6a06" containerName="cleanup" Oct 30 23:02:05.594495 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:05.594342 6006 cpu_manager.go:414] "containerMap: removing container" podUID="9b620ecf-2617-455e-af1f-27286b844486" containerName="revoke-keys" Oct 30 23:02:05.594495 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:05.594350 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b620ecf-2617-455e-af1f-27286b844486" containerName="revoke-keys" Oct 30 23:02:05.594614 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:05.594598 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="170acbb8-9b1e-451f-a812-4390a7cc6a06" containerName="cleanup" Oct 30 23:02:05.594670 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:05.594618 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b620ecf-2617-455e-af1f-27286b844486" containerName="revoke-keys" Oct 30 23:02:05.600583 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:05.600552 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-55fz5" Oct 30 23:02:05.621263 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:05.621220 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6338719f-b0e9-4ac2-9b60-d4111f5e2678-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-55fz5\" (UID: \"6338719f-b0e9-4ac2-9b60-d4111f5e2678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-55fz5" Oct 30 23:02:05.621408 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:05.621381 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6338719f-b0e9-4ac2-9b60-d4111f5e2678-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-55fz5\" (UID: \"6338719f-b0e9-4ac2-9b60-d4111f5e2678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-55fz5" Oct 30 23:02:05.621453 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:05.621425 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2wps\" (UniqueName: \"kubernetes.io/projected/6338719f-b0e9-4ac2-9b60-d4111f5e2678-kube-api-access-r2wps\") pod \"maas-api-revoke-sub-f03629db-55fz5\" (UID: \"6338719f-b0e9-4ac2-9b60-d4111f5e2678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-55fz5" Oct 30 23:02:05.650546 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:05.650514 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-55fz5"] Oct 30 23:02:05.695478 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:05.695439 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:02:05.695478 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:05.695460 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k2h55\"" Oct 30 23:02:05.695699 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:05.695487 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 23:02:05.696909 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:05.696887 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 23:02:05.722450 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:05.722414 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6338719f-b0e9-4ac2-9b60-d4111f5e2678-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-55fz5\" (UID: \"6338719f-b0e9-4ac2-9b60-d4111f5e2678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-55fz5" Oct 30 23:02:05.722628 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:05.722473 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2wps\" (UniqueName: \"kubernetes.io/projected/6338719f-b0e9-4ac2-9b60-d4111f5e2678-kube-api-access-r2wps\") pod \"maas-api-revoke-sub-f03629db-55fz5\" (UID: \"6338719f-b0e9-4ac2-9b60-d4111f5e2678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-55fz5" Oct 30 23:02:05.722628 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:05.722566 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6338719f-b0e9-4ac2-9b60-d4111f5e2678-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-55fz5\" (UID: \"6338719f-b0e9-4ac2-9b60-d4111f5e2678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-55fz5" Oct 30 23:02:05.723294 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:05.723241 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6338719f-b0e9-4ac2-9b60-d4111f5e2678-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-55fz5\" (UID: \"6338719f-b0e9-4ac2-9b60-d4111f5e2678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-55fz5" Oct 30 23:02:05.725023 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:05.724998 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6338719f-b0e9-4ac2-9b60-d4111f5e2678-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-55fz5\" (UID: \"6338719f-b0e9-4ac2-9b60-d4111f5e2678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-55fz5" Oct 30 23:02:05.783536 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:05.783469 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2wps\" (UniqueName: \"kubernetes.io/projected/6338719f-b0e9-4ac2-9b60-d4111f5e2678-kube-api-access-r2wps\") pod \"maas-api-revoke-sub-f03629db-55fz5\" (UID: \"6338719f-b0e9-4ac2-9b60-d4111f5e2678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-55fz5" Oct 30 23:02:05.914381 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:05.914291 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-55fz5" Oct 30 23:02:06.298239 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:06.298209 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-55fz5"] Oct 30 23:02:06.300300 ip-10-0-135-7 kubenswrapper[6006]: W1030 23:02:06.300268 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6338719f_b0e9_4ac2_9b60_d4111f5e2678.slice/crio-36f1ae8621a08124bf7801a1bad29b44c4aafbc3818a58ecc201e93009934346 WatchSource:0}: Error finding container 36f1ae8621a08124bf7801a1bad29b44c4aafbc3818a58ecc201e93009934346: Status 404 returned error can't find the container with id 36f1ae8621a08124bf7801a1bad29b44c4aafbc3818a58ecc201e93009934346 Oct 30 23:02:06.670346 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:06.670307 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-55fz5" event={"ID":"6338719f-b0e9-4ac2-9b60-d4111f5e2678","Type":"ContainerStarted","Data":"8401e2b11176a6e1a17674a3368dc2e5c96300d7997fbabdd15e3a38404843ac"} Oct 30 23:02:06.670782 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:06.670364 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-55fz5" event={"ID":"6338719f-b0e9-4ac2-9b60-d4111f5e2678","Type":"ContainerStarted","Data":"36f1ae8621a08124bf7801a1bad29b44c4aafbc3818a58ecc201e93009934346"} Oct 30 23:02:06.784079 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:06.784014 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-55fz5" podStartSLOduration=1.783996157 podStartE2EDuration="1.783996157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:02:06.783487998 +0000 UTC m=+4329.335664299" watchObservedRunningTime="2026-10-30 23:02:06.783996157 +0000 UTC m=+4329.336172454" Oct 30 23:02:07.677156 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:07.677118 6006 generic.go:356] "Generic (PLEG): container finished" podID="6338719f-b0e9-4ac2-9b60-d4111f5e2678" containerID="8401e2b11176a6e1a17674a3368dc2e5c96300d7997fbabdd15e3a38404843ac" exitCode=0 Oct 30 23:02:07.677592 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:07.677170 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-55fz5" event={"ID":"6338719f-b0e9-4ac2-9b60-d4111f5e2678","Type":"ContainerDied","Data":"8401e2b11176a6e1a17674a3368dc2e5c96300d7997fbabdd15e3a38404843ac"} Oct 30 23:02:08.833493 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:08.833467 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-55fz5" Oct 30 23:02:08.848445 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:08.848411 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6338719f-b0e9-4ac2-9b60-d4111f5e2678-kube-api-access-r2wps\" (UniqueName: \"kubernetes.io/projected/6338719f-b0e9-4ac2-9b60-d4111f5e2678-kube-api-access-r2wps\") pod \"6338719f-b0e9-4ac2-9b60-d4111f5e2678\" (UID: \"6338719f-b0e9-4ac2-9b60-d4111f5e2678\") " Oct 30 23:02:08.848598 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:08.848499 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6338719f-b0e9-4ac2-9b60-d4111f5e2678-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6338719f-b0e9-4ac2-9b60-d4111f5e2678-maas-api-xks-ca\") pod \"6338719f-b0e9-4ac2-9b60-d4111f5e2678\" (UID: \"6338719f-b0e9-4ac2-9b60-d4111f5e2678\") " Oct 30 23:02:08.848598 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:08.848573 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6338719f-b0e9-4ac2-9b60-d4111f5e2678-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6338719f-b0e9-4ac2-9b60-d4111f5e2678-maas-api-openshift-ca\") pod \"6338719f-b0e9-4ac2-9b60-d4111f5e2678\" (UID: \"6338719f-b0e9-4ac2-9b60-d4111f5e2678\") " Oct 30 23:02:08.848952 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:08.848931 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6338719f-b0e9-4ac2-9b60-d4111f5e2678-maas-api-openshift-ca" pod "6338719f-b0e9-4ac2-9b60-d4111f5e2678" (UID: "6338719f-b0e9-4ac2-9b60-d4111f5e2678"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:02:08.850530 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:08.850505 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6338719f-b0e9-4ac2-9b60-d4111f5e2678-maas-api-xks-ca" pod "6338719f-b0e9-4ac2-9b60-d4111f5e2678" (UID: "6338719f-b0e9-4ac2-9b60-d4111f5e2678"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:02:08.850635 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:08.850593 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6338719f-b0e9-4ac2-9b60-d4111f5e2678-kube-api-access-r2wps" pod "6338719f-b0e9-4ac2-9b60-d4111f5e2678" (UID: "6338719f-b0e9-4ac2-9b60-d4111f5e2678"). InnerVolumeSpecName "kube-api-access-r2wps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:02:08.949814 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:08.949718 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6338719f-b0e9-4ac2-9b60-d4111f5e2678-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:02:08.949814 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:08.949757 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6338719f-b0e9-4ac2-9b60-d4111f5e2678-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:02:08.949814 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:08.949772 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2wps\" (UniqueName: \"kubernetes.io/projected/6338719f-b0e9-4ac2-9b60-d4111f5e2678-kube-api-access-r2wps\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:02:09.688462 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:09.688425 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-55fz5" Oct 30 23:02:09.688462 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:09.688455 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-55fz5" event={"ID":"6338719f-b0e9-4ac2-9b60-d4111f5e2678","Type":"ContainerDied","Data":"36f1ae8621a08124bf7801a1bad29b44c4aafbc3818a58ecc201e93009934346"} Oct 30 23:02:09.688707 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:09.688505 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36f1ae8621a08124bf7801a1bad29b44c4aafbc3818a58ecc201e93009934346" Oct 30 23:02:18.468657 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:18.468616 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 23:02:26.537152 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:26.537108 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bpzbs"] Oct 30 23:02:26.537593 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:26.537572 6006 cpu_manager.go:414] "containerMap: removing container" podUID="6338719f-b0e9-4ac2-9b60-d4111f5e2678" containerName="revoke-keys" Oct 30 23:02:26.537593 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:26.537583 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6338719f-b0e9-4ac2-9b60-d4111f5e2678" containerName="revoke-keys" Oct 30 23:02:26.537674 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:26.537663 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="6338719f-b0e9-4ac2-9b60-d4111f5e2678" containerName="revoke-keys" Oct 30 23:02:26.541047 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:26.541027 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bpzbs" Oct 30 23:02:26.590666 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:26.590631 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:02:26.590666 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:26.590669 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k2h55\"" Oct 30 23:02:26.637370 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:26.637335 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 23:02:26.637370 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:26.637366 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 23:02:26.639049 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:26.639024 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bpzbs"] Oct 30 23:02:26.670859 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:26.670817 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b49tb\" (UniqueName: \"kubernetes.io/projected/0ef148fd-ee55-4894-b6c8-347432b24308-kube-api-access-b49tb\") pod \"maas-api-revoke-sub-0165d7d2-bpzbs\" (UID: \"0ef148fd-ee55-4894-b6c8-347432b24308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bpzbs" Oct 30 23:02:26.671033 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:26.670961 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ef148fd-ee55-4894-b6c8-347432b24308-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-bpzbs\" (UID: \"0ef148fd-ee55-4894-b6c8-347432b24308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bpzbs" Oct 30 23:02:26.671428 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:26.671062 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ef148fd-ee55-4894-b6c8-347432b24308-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-bpzbs\" (UID: \"0ef148fd-ee55-4894-b6c8-347432b24308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bpzbs" Oct 30 23:02:26.772741 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:26.772704 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b49tb\" (UniqueName: \"kubernetes.io/projected/0ef148fd-ee55-4894-b6c8-347432b24308-kube-api-access-b49tb\") pod \"maas-api-revoke-sub-0165d7d2-bpzbs\" (UID: \"0ef148fd-ee55-4894-b6c8-347432b24308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bpzbs" Oct 30 23:02:26.772932 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:26.772756 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ef148fd-ee55-4894-b6c8-347432b24308-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-bpzbs\" (UID: \"0ef148fd-ee55-4894-b6c8-347432b24308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bpzbs" Oct 30 23:02:26.772932 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:26.772804 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ef148fd-ee55-4894-b6c8-347432b24308-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-bpzbs\" (UID: \"0ef148fd-ee55-4894-b6c8-347432b24308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bpzbs" Oct 30 23:02:26.773494 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:26.773475 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ef148fd-ee55-4894-b6c8-347432b24308-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-bpzbs\" (UID: \"0ef148fd-ee55-4894-b6c8-347432b24308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bpzbs" Oct 30 23:02:26.775223 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:26.775196 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ef148fd-ee55-4894-b6c8-347432b24308-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-bpzbs\" (UID: \"0ef148fd-ee55-4894-b6c8-347432b24308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bpzbs" Oct 30 23:02:26.831226 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:26.831192 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b49tb\" (UniqueName: \"kubernetes.io/projected/0ef148fd-ee55-4894-b6c8-347432b24308-kube-api-access-b49tb\") pod \"maas-api-revoke-sub-0165d7d2-bpzbs\" (UID: \"0ef148fd-ee55-4894-b6c8-347432b24308\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bpzbs" Oct 30 23:02:26.853966 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:26.853935 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bpzbs" Oct 30 23:02:27.225229 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:27.225197 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bpzbs"] Oct 30 23:02:27.228085 ip-10-0-135-7 kubenswrapper[6006]: W1030 23:02:27.228051 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef148fd_ee55_4894_b6c8_347432b24308.slice/crio-23eca468e6efd1925880bf33aa3246b92403466ddad8fbc9e2c789a3b1af294d WatchSource:0}: Error finding container 23eca468e6efd1925880bf33aa3246b92403466ddad8fbc9e2c789a3b1af294d: Status 404 returned error can't find the container with id 23eca468e6efd1925880bf33aa3246b92403466ddad8fbc9e2c789a3b1af294d Oct 30 23:02:27.791464 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:27.791427 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bpzbs" event={"ID":"0ef148fd-ee55-4894-b6c8-347432b24308","Type":"ContainerStarted","Data":"2ad9c58dd9faa9c3d3755e049aeae0022fafe9541d9c2c2680c647836038e1c9"} Oct 30 23:02:27.791927 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:27.791493 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bpzbs" event={"ID":"0ef148fd-ee55-4894-b6c8-347432b24308","Type":"ContainerStarted","Data":"23eca468e6efd1925880bf33aa3246b92403466ddad8fbc9e2c789a3b1af294d"} Oct 30 23:02:27.901030 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:27.900965 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bpzbs" podStartSLOduration=1.9009463009999998 podStartE2EDuration="1.900946301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:02:27.900752374 +0000 UTC m=+4350.452928677" watchObservedRunningTime="2026-10-30 23:02:27.900946301 +0000 UTC m=+4350.453122602" Oct 30 23:02:27.911144 ip-10-0-135-7 kubenswrapper[6006]: E1030 23:02:27.911101 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 23:02:28.798806 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:28.798769 6006 generic.go:356] "Generic (PLEG): container finished" podID="0ef148fd-ee55-4894-b6c8-347432b24308" containerID="2ad9c58dd9faa9c3d3755e049aeae0022fafe9541d9c2c2680c647836038e1c9" exitCode=0 Oct 30 23:02:28.799210 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:28.798865 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bpzbs" event={"ID":"0ef148fd-ee55-4894-b6c8-347432b24308","Type":"ContainerDied","Data":"2ad9c58dd9faa9c3d3755e049aeae0022fafe9541d9c2c2680c647836038e1c9"} Oct 30 23:02:29.950567 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:29.950537 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bpzbs" Oct 30 23:02:30.108076 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:30.107980 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ef148fd-ee55-4894-b6c8-347432b24308-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ef148fd-ee55-4894-b6c8-347432b24308-maas-api-openshift-ca\") pod \"0ef148fd-ee55-4894-b6c8-347432b24308\" (UID: \"0ef148fd-ee55-4894-b6c8-347432b24308\") " Oct 30 23:02:30.108278 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:30.108136 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ef148fd-ee55-4894-b6c8-347432b24308-kube-api-access-b49tb\" (UniqueName: \"kubernetes.io/projected/0ef148fd-ee55-4894-b6c8-347432b24308-kube-api-access-b49tb\") pod \"0ef148fd-ee55-4894-b6c8-347432b24308\" (UID: \"0ef148fd-ee55-4894-b6c8-347432b24308\") " Oct 30 23:02:30.108278 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:30.108221 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ef148fd-ee55-4894-b6c8-347432b24308-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ef148fd-ee55-4894-b6c8-347432b24308-maas-api-xks-ca\") pod \"0ef148fd-ee55-4894-b6c8-347432b24308\" (UID: \"0ef148fd-ee55-4894-b6c8-347432b24308\") " Oct 30 23:02:30.108517 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:30.108484 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ef148fd-ee55-4894-b6c8-347432b24308-maas-api-openshift-ca" pod "0ef148fd-ee55-4894-b6c8-347432b24308" (UID: "0ef148fd-ee55-4894-b6c8-347432b24308"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:02:30.110550 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:30.110522 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ef148fd-ee55-4894-b6c8-347432b24308-maas-api-xks-ca" pod "0ef148fd-ee55-4894-b6c8-347432b24308" (UID: "0ef148fd-ee55-4894-b6c8-347432b24308"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:02:30.110784 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:30.110734 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ef148fd-ee55-4894-b6c8-347432b24308-kube-api-access-b49tb" pod "0ef148fd-ee55-4894-b6c8-347432b24308" (UID: "0ef148fd-ee55-4894-b6c8-347432b24308"). InnerVolumeSpecName "kube-api-access-b49tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:02:30.209991 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:30.209953 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ef148fd-ee55-4894-b6c8-347432b24308-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:02:30.209991 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:30.209984 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b49tb\" (UniqueName: \"kubernetes.io/projected/0ef148fd-ee55-4894-b6c8-347432b24308-kube-api-access-b49tb\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:02:30.209991 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:30.209996 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ef148fd-ee55-4894-b6c8-347432b24308-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:02:30.810765 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:30.810732 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bpzbs" Oct 30 23:02:30.810765 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:30.810750 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bpzbs" event={"ID":"0ef148fd-ee55-4894-b6c8-347432b24308","Type":"ContainerDied","Data":"23eca468e6efd1925880bf33aa3246b92403466ddad8fbc9e2c789a3b1af294d"} Oct 30 23:02:30.810993 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:30.810793 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23eca468e6efd1925880bf33aa3246b92403466ddad8fbc9e2c789a3b1af294d" Oct 30 23:02:38.295451 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:38.295408 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbfh2"] Oct 30 23:02:38.295893 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:38.295821 6006 cpu_manager.go:414] "containerMap: removing container" podUID="0ef148fd-ee55-4894-b6c8-347432b24308" containerName="revoke-keys" Oct 30 23:02:38.295893 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:38.295833 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ef148fd-ee55-4894-b6c8-347432b24308" containerName="revoke-keys" Oct 30 23:02:38.295973 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:38.295932 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ef148fd-ee55-4894-b6c8-347432b24308" containerName="revoke-keys" Oct 30 23:02:38.299188 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:38.299162 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbfh2" Oct 30 23:02:38.348653 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:38.348615 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:02:38.348854 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:38.348662 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 23:02:38.348854 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:38.348696 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 23:02:38.386449 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:38.386391 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxbzx\" (UniqueName: \"kubernetes.io/projected/4d269f45-f2db-48c2-b8ce-63f0476a97f9-kube-api-access-qxbzx\") pod \"maas-api-revoke-sub-6b19338f-nbfh2\" (UID: \"4d269f45-f2db-48c2-b8ce-63f0476a97f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbfh2" Oct 30 23:02:38.386449 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:38.386445 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d269f45-f2db-48c2-b8ce-63f0476a97f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-nbfh2\" (UID: \"4d269f45-f2db-48c2-b8ce-63f0476a97f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbfh2" Oct 30 23:02:38.386734 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:38.386592 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d269f45-f2db-48c2-b8ce-63f0476a97f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-nbfh2\" (UID: \"4d269f45-f2db-48c2-b8ce-63f0476a97f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbfh2" Oct 30 23:02:38.395414 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:38.395380 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k2h55\"" Oct 30 23:02:38.397751 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:38.397721 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbfh2"] Oct 30 23:02:38.487961 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:38.487920 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxbzx\" (UniqueName: \"kubernetes.io/projected/4d269f45-f2db-48c2-b8ce-63f0476a97f9-kube-api-access-qxbzx\") pod \"maas-api-revoke-sub-6b19338f-nbfh2\" (UID: \"4d269f45-f2db-48c2-b8ce-63f0476a97f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbfh2" Oct 30 23:02:38.487961 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:38.487963 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d269f45-f2db-48c2-b8ce-63f0476a97f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-nbfh2\" (UID: \"4d269f45-f2db-48c2-b8ce-63f0476a97f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbfh2" Oct 30 23:02:38.488226 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:38.488010 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d269f45-f2db-48c2-b8ce-63f0476a97f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-nbfh2\" (UID: \"4d269f45-f2db-48c2-b8ce-63f0476a97f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbfh2" Oct 30 23:02:38.488726 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:38.488696 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d269f45-f2db-48c2-b8ce-63f0476a97f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-nbfh2\" (UID: \"4d269f45-f2db-48c2-b8ce-63f0476a97f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbfh2" Oct 30 23:02:38.490579 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:38.490556 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d269f45-f2db-48c2-b8ce-63f0476a97f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-nbfh2\" (UID: \"4d269f45-f2db-48c2-b8ce-63f0476a97f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbfh2" Oct 30 23:02:38.547019 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:38.546945 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxbzx\" (UniqueName: \"kubernetes.io/projected/4d269f45-f2db-48c2-b8ce-63f0476a97f9-kube-api-access-qxbzx\") pod \"maas-api-revoke-sub-6b19338f-nbfh2\" (UID: \"4d269f45-f2db-48c2-b8ce-63f0476a97f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbfh2" Oct 30 23:02:38.615022 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:38.614985 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbfh2" Oct 30 23:02:38.983505 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:38.983472 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbfh2"] Oct 30 23:02:38.986376 ip-10-0-135-7 kubenswrapper[6006]: W1030 23:02:38.986348 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d269f45_f2db_48c2_b8ce_63f0476a97f9.slice/crio-1647aefe1be1eeed9526e6ef5db7a0890b2d3f097437a7b8c37e4f96485ac0d5 WatchSource:0}: Error finding container 1647aefe1be1eeed9526e6ef5db7a0890b2d3f097437a7b8c37e4f96485ac0d5: Status 404 returned error can't find the container with id 1647aefe1be1eeed9526e6ef5db7a0890b2d3f097437a7b8c37e4f96485ac0d5 Oct 30 23:02:39.857793 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:39.857752 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbfh2" event={"ID":"4d269f45-f2db-48c2-b8ce-63f0476a97f9","Type":"ContainerStarted","Data":"8abe7b233644be6ca2d8d95254726ffe43f4202b0619158d87114e0605dde9ef"} Oct 30 23:02:39.858230 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:39.857821 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbfh2" event={"ID":"4d269f45-f2db-48c2-b8ce-63f0476a97f9","Type":"ContainerStarted","Data":"1647aefe1be1eeed9526e6ef5db7a0890b2d3f097437a7b8c37e4f96485ac0d5"} Oct 30 23:02:39.976215 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:39.976158 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbfh2" podStartSLOduration=1.976144209 podStartE2EDuration="1.976144209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:02:39.975649286 +0000 UTC m=+4362.527825586" watchObservedRunningTime="2026-10-30 23:02:39.976144209 +0000 UTC m=+4362.528320566" Oct 30 23:02:40.867502 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:40.867465 6006 generic.go:356] "Generic (PLEG): container finished" podID="4d269f45-f2db-48c2-b8ce-63f0476a97f9" containerID="8abe7b233644be6ca2d8d95254726ffe43f4202b0619158d87114e0605dde9ef" exitCode=0 Oct 30 23:02:40.867946 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:40.867548 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbfh2" event={"ID":"4d269f45-f2db-48c2-b8ce-63f0476a97f9","Type":"ContainerDied","Data":"8abe7b233644be6ca2d8d95254726ffe43f4202b0619158d87114e0605dde9ef"} Oct 30 23:02:42.023321 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:42.023291 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbfh2" Oct 30 23:02:42.126510 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:42.126407 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d269f45-f2db-48c2-b8ce-63f0476a97f9-kube-api-access-qxbzx\" (UniqueName: \"kubernetes.io/projected/4d269f45-f2db-48c2-b8ce-63f0476a97f9-kube-api-access-qxbzx\") pod \"4d269f45-f2db-48c2-b8ce-63f0476a97f9\" (UID: \"4d269f45-f2db-48c2-b8ce-63f0476a97f9\") " Oct 30 23:02:42.126510 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:42.126495 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d269f45-f2db-48c2-b8ce-63f0476a97f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d269f45-f2db-48c2-b8ce-63f0476a97f9-maas-api-xks-ca\") pod \"4d269f45-f2db-48c2-b8ce-63f0476a97f9\" (UID: \"4d269f45-f2db-48c2-b8ce-63f0476a97f9\") " Oct 30 23:02:42.126834 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:42.126529 6006 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d269f45-f2db-48c2-b8ce-63f0476a97f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d269f45-f2db-48c2-b8ce-63f0476a97f9-maas-api-openshift-ca\") pod \"4d269f45-f2db-48c2-b8ce-63f0476a97f9\" (UID: \"4d269f45-f2db-48c2-b8ce-63f0476a97f9\") " Oct 30 23:02:42.126935 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:42.126900 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d269f45-f2db-48c2-b8ce-63f0476a97f9-maas-api-openshift-ca" pod "4d269f45-f2db-48c2-b8ce-63f0476a97f9" (UID: "4d269f45-f2db-48c2-b8ce-63f0476a97f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:02:42.128661 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:42.128626 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d269f45-f2db-48c2-b8ce-63f0476a97f9-maas-api-xks-ca" pod "4d269f45-f2db-48c2-b8ce-63f0476a97f9" (UID: "4d269f45-f2db-48c2-b8ce-63f0476a97f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:02:42.128772 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:42.128690 6006 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d269f45-f2db-48c2-b8ce-63f0476a97f9-kube-api-access-qxbzx" pod "4d269f45-f2db-48c2-b8ce-63f0476a97f9" (UID: "4d269f45-f2db-48c2-b8ce-63f0476a97f9"). InnerVolumeSpecName "kube-api-access-qxbzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:02:42.228224 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:42.228180 6006 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxbzx\" (UniqueName: \"kubernetes.io/projected/4d269f45-f2db-48c2-b8ce-63f0476a97f9-kube-api-access-qxbzx\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:02:42.228224 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:42.228216 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d269f45-f2db-48c2-b8ce-63f0476a97f9-maas-api-xks-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:02:42.228224 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:42.228227 6006 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d269f45-f2db-48c2-b8ce-63f0476a97f9-maas-api-openshift-ca\") on node \"ip-10-0-135-7.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:02:42.879621 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:42.879591 6006 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbfh2" Oct 30 23:02:42.879791 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:42.879629 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbfh2" event={"ID":"4d269f45-f2db-48c2-b8ce-63f0476a97f9","Type":"ContainerDied","Data":"1647aefe1be1eeed9526e6ef5db7a0890b2d3f097437a7b8c37e4f96485ac0d5"} Oct 30 23:02:42.879791 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:42.879681 6006 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1647aefe1be1eeed9526e6ef5db7a0890b2d3f097437a7b8c37e4f96485ac0d5" Oct 30 23:02:50.603039 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:50.602996 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-667cc846fb-7smlq_c328f12f-dbd3-4d02-b759-08a11ab99993/authorino/0.log" Oct 30 23:02:51.039223 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:02:51.039124 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p76jm"] Oct 30 23:02:57.910632 ip-10-0-135-7 kubenswrapper[6006]: E1030 23:02:57.910595 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 23:03:02.061527 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:02.061493 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-68ns6_f001662b-cac0-4ac1-a34f-d6bc732ee69f/manager/0.log" Oct 30 23:03:02.351503 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:02.351406 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-xzv68_15211cfe-5655-448d-b764-ae4de94fecb8/manager/0.log" Oct 30 23:03:02.672445 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:02.672364 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-4n2tx_5e451335-7336-48fb-b23c-f8b30de7fd33/manager/0.log" Oct 30 23:03:03.826813 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:03.826782 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-bg2f2_d46d3bef-fb7d-42c2-82ed-61a009a2c713/manager/0.log" Oct 30 23:03:05.443344 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:05.443308 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-bpzbs_0ef148fd-ee55-4894-b6c8-347432b24308/revoke-keys/0.log" Oct 30 23:03:05.677325 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:05.677288 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-nbfh2_4d269f45-f2db-48c2-b8ce-63f0476a97f9/revoke-keys/0.log" Oct 30 23:03:05.919862 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:05.919833 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-55fz5_6338719f-b0e9-4ac2-9b60-d4111f5e2678/revoke-keys/0.log" Oct 30 23:03:07.770400 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:07.770357 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-667cc846fb-7smlq_c328f12f-dbd3-4d02-b759-08a11ab99993/authorino/0.log" Oct 30 23:03:09.460772 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:09.460731 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-p76jm_47477b2f-5e3f-437c-8203-80aac978e56a/limitador/0.log" Oct 30 23:03:09.691595 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:09.691561 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-7gcxz_e7333812-2624-427a-ac27-7f953e3d517e/manager/0.log" Oct 30 23:03:10.686597 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:10.686566 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-65844c896fp8nft_e592cd54-dc62-473d-9993-976186532144/istio-proxy/0.log" Oct 30 23:03:10.967802 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:10.967718 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-kv5vn_64dd55d9-83da-4513-b49a-fbca9fc1cd9b/discovery/0.log" Oct 30 23:03:11.984274 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:11.984220 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-6cmps_2c3b94ab-018e-480f-b12c-faa5544fc2e1/istio-proxy/0.log" Oct 30 23:03:12.587758 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:12.587726 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-ck8bj_33b53342-a5ab-459b-8005-da1267a7c0b4/istio-proxy/0.log" Oct 30 23:03:13.146986 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:13.146950 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-jst4g_82a12523-b466-4041-bb2a-73db5538706c/istio-proxy/0.log" Oct 30 23:03:14.417426 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:14.417389 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4_989395fe-e528-4e51-96aa-e4b2a89e3b6f/istio-proxy/0.log" Oct 30 23:03:15.123664 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:15.123629 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76684dbbf-8mwh7_11ba3d2b-2d12-458c-9984-1dd82c275e10/router/0.log" Oct 30 23:03:18.609815 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:18.609782 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-667cc846fb-7smlq_c328f12f-dbd3-4d02-b759-08a11ab99993/authorino/0.log" Oct 30 23:03:27.909899 ip-10-0-135-7 kubenswrapper[6006]: E1030 23:03:27.909860 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 23:03:28.502428 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:28.502379 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-68ns6_f001662b-cac0-4ac1-a34f-d6bc732ee69f/manager/0.log" Oct 30 23:03:28.783125 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:28.783035 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-xzv68_15211cfe-5655-448d-b764-ae4de94fecb8/manager/0.log" Oct 30 23:03:29.115895 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:29.115856 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-4n2tx_5e451335-7336-48fb-b23c-f8b30de7fd33/manager/0.log" Oct 30 23:03:30.029614 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:30.029566 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-bg2f2_d46d3bef-fb7d-42c2-82ed-61a009a2c713/manager/0.log" Oct 30 23:03:31.475039 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:31.475008 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-bpzbs_0ef148fd-ee55-4894-b6c8-347432b24308/revoke-keys/0.log" Oct 30 23:03:31.691855 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:31.691817 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-nbfh2_4d269f45-f2db-48c2-b8ce-63f0476a97f9/revoke-keys/0.log" Oct 30 23:03:31.914780 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:31.914752 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-55fz5_6338719f-b0e9-4ac2-9b60-d4111f5e2678/revoke-keys/0.log" Oct 30 23:03:33.367508 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:33.367475 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-667cc846fb-7smlq_c328f12f-dbd3-4d02-b759-08a11ab99993/authorino/0.log" Oct 30 23:03:34.919930 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:34.919903 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-p76jm_47477b2f-5e3f-437c-8203-80aac978e56a/limitador/0.log" Oct 30 23:03:35.155019 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:35.154923 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-7gcxz_e7333812-2624-427a-ac27-7f953e3d517e/manager/0.log" Oct 30 23:03:35.761088 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:35.760995 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-65844c896fp8nft_e592cd54-dc62-473d-9993-976186532144/istio-proxy/0.log" Oct 30 23:03:36.011731 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:36.011634 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-kv5vn_64dd55d9-83da-4513-b49a-fbca9fc1cd9b/discovery/0.log" Oct 30 23:03:36.982201 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:36.982173 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-6cmps_2c3b94ab-018e-480f-b12c-faa5544fc2e1/istio-proxy/0.log" Oct 30 23:03:37.518624 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:37.518582 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-ck8bj_33b53342-a5ab-459b-8005-da1267a7c0b4/istio-proxy/0.log" Oct 30 23:03:38.100106 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:38.100063 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-jst4g_82a12523-b466-4041-bb2a-73db5538706c/istio-proxy/0.log" Oct 30 23:03:39.340791 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:39.340757 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-744fc677d7-72st4_989395fe-e528-4e51-96aa-e4b2a89e3b6f/istio-proxy/0.log" Oct 30 23:03:40.085317 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:03:40.085285 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76684dbbf-8mwh7_11ba3d2b-2d12-458c-9984-1dd82c275e10/router/0.log" Oct 30 23:03:57.910023 ip-10-0-135-7 kubenswrapper[6006]: E1030 23:03:57.909982 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 23:04:27.910921 ip-10-0-135-7 kubenswrapper[6006]: E1030 23:04:27.910878 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 23:04:57.910332 ip-10-0-135-7 kubenswrapper[6006]: E1030 23:04:57.910291 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 23:04:58.312372 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:04:58.312341 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 23:04:58.355770 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:04:58.355742 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8w8b5_5fcd0d3b-74a6-44c8-915a-c3e8fd25ed51/console-operator/1.log" Oct 30 23:05:00.303499 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:00.303467 6006 scope.go:122] "RemoveContainer" containerID="a5b4b0cb81079c931e6a507b88bf0bf2a5ef6fe17e88fc1917032dc4c4339fcf" Oct 30 23:05:05.798180 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:05.798089 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lfq7z_1d430f7e-00f1-4b9e-9427-3e9971cfff31/global-pull-secret-syncer/0.log" Oct 30 23:05:06.397177 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:06.397143 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-2xctf_8bf0fd8f-2294-4729-9f49-a151de8627aa/connection-checker/0.log" Oct 30 23:05:06.823512 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:06.823478 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6pjx4_0dce5b38-e1a7-41e3-8100-96af4e3c2b06/connection-checker/0.log" Oct 30 23:05:07.235385 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:07.235293 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-k6p2v_9df43c88-a01f-4ec7-a385-d5afcf6f9e6c/connection-checker/0.log" Oct 30 23:05:08.178932 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:08.178900 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-929rz_64086350-55f6-4da5-a871-3f23c1755371/konnectivity-agent/0.log" Oct 30 23:05:09.429701 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:09.429660 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-7.us-west-2.compute.internal_774172d5b7cfcf0d4b41c09fc8c830a1/haproxy/0.log" Oct 30 23:05:14.643367 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:14.643329 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-667cc846fb-7smlq_c328f12f-dbd3-4d02-b759-08a11ab99993/authorino/0.log" Oct 30 23:05:14.996214 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:14.996125 6006 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2cfz8/perf-node-gather-daemonset-kmsxc"] Oct 30 23:05:14.996578 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:14.996565 6006 cpu_manager.go:414] "containerMap: removing container" podUID="4d269f45-f2db-48c2-b8ce-63f0476a97f9" containerName="revoke-keys" Oct 30 23:05:14.996636 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:14.996579 6006 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d269f45-f2db-48c2-b8ce-63f0476a97f9" containerName="revoke-keys" Oct 30 23:05:14.996678 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:14.996649 6006 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d269f45-f2db-48c2-b8ce-63f0476a97f9" containerName="revoke-keys" Oct 30 23:05:14.999899 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:14.999875 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-kmsxc" Oct 30 23:05:15.049297 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:15.049231 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2cfz8\"/\"openshift-service-ca.crt\"" Oct 30 23:05:15.068812 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:15.068770 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aae40325-6969-414a-b615-58d57b5a9e35-sys\") pod \"perf-node-gather-daemonset-kmsxc\" (UID: \"aae40325-6969-414a-b615-58d57b5a9e35\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-kmsxc" Oct 30 23:05:15.069012 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:15.068862 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aae40325-6969-414a-b615-58d57b5a9e35-lib-modules\") pod \"perf-node-gather-daemonset-kmsxc\" (UID: \"aae40325-6969-414a-b615-58d57b5a9e35\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-kmsxc" Oct 30 23:05:15.069012 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:15.068959 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkkx9\" (UniqueName: \"kubernetes.io/projected/aae40325-6969-414a-b615-58d57b5a9e35-kube-api-access-hkkx9\") pod \"perf-node-gather-daemonset-kmsxc\" (UID: \"aae40325-6969-414a-b615-58d57b5a9e35\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-kmsxc" Oct 30 23:05:15.069111 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:15.069091 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aae40325-6969-414a-b615-58d57b5a9e35-proc\") pod \"perf-node-gather-daemonset-kmsxc\" (UID: \"aae40325-6969-414a-b615-58d57b5a9e35\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-kmsxc" Oct 30 23:05:15.069154 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:15.069123 6006 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aae40325-6969-414a-b615-58d57b5a9e35-podres\") pod \"perf-node-gather-daemonset-kmsxc\" (UID: \"aae40325-6969-414a-b615-58d57b5a9e35\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-kmsxc" Oct 30 23:05:15.095910 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:15.095875 6006 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2cfz8\"/\"kube-root-ca.crt\"" Oct 30 23:05:15.096119 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:15.096003 6006 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2cfz8\"/\"default-dockercfg-4lpsw\"" Oct 30 23:05:15.097735 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:15.097707 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2cfz8/perf-node-gather-daemonset-kmsxc"] Oct 30 23:05:15.170216 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:15.170167 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aae40325-6969-414a-b615-58d57b5a9e35-proc\") pod \"perf-node-gather-daemonset-kmsxc\" (UID: \"aae40325-6969-414a-b615-58d57b5a9e35\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-kmsxc" Oct 30 23:05:15.170216 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:15.170219 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aae40325-6969-414a-b615-58d57b5a9e35-podres\") pod \"perf-node-gather-daemonset-kmsxc\" (UID: \"aae40325-6969-414a-b615-58d57b5a9e35\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-kmsxc" Oct 30 23:05:15.170512 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:15.170283 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aae40325-6969-414a-b615-58d57b5a9e35-sys\") pod \"perf-node-gather-daemonset-kmsxc\" (UID: \"aae40325-6969-414a-b615-58d57b5a9e35\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-kmsxc" Oct 30 23:05:15.170512 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:15.170349 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aae40325-6969-414a-b615-58d57b5a9e35-proc\") pod \"perf-node-gather-daemonset-kmsxc\" (UID: \"aae40325-6969-414a-b615-58d57b5a9e35\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-kmsxc" Oct 30 23:05:15.170512 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:15.170366 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aae40325-6969-414a-b615-58d57b5a9e35-lib-modules\") pod \"perf-node-gather-daemonset-kmsxc\" (UID: \"aae40325-6969-414a-b615-58d57b5a9e35\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-kmsxc" Oct 30 23:05:15.170512 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:15.170361 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aae40325-6969-414a-b615-58d57b5a9e35-podres\") pod \"perf-node-gather-daemonset-kmsxc\" (UID: \"aae40325-6969-414a-b615-58d57b5a9e35\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-kmsxc" Oct 30 23:05:15.170512 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:15.170401 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aae40325-6969-414a-b615-58d57b5a9e35-sys\") pod \"perf-node-gather-daemonset-kmsxc\" (UID: \"aae40325-6969-414a-b615-58d57b5a9e35\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-kmsxc" Oct 30 23:05:15.170512 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:15.170438 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aae40325-6969-414a-b615-58d57b5a9e35-lib-modules\") pod \"perf-node-gather-daemonset-kmsxc\" (UID: \"aae40325-6969-414a-b615-58d57b5a9e35\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-kmsxc" Oct 30 23:05:15.170512 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:15.170502 6006 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkkx9\" (UniqueName: \"kubernetes.io/projected/aae40325-6969-414a-b615-58d57b5a9e35-kube-api-access-hkkx9\") pod \"perf-node-gather-daemonset-kmsxc\" (UID: \"aae40325-6969-414a-b615-58d57b5a9e35\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-kmsxc" Oct 30 23:05:15.229747 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:15.229718 6006 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkkx9\" (UniqueName: \"kubernetes.io/projected/aae40325-6969-414a-b615-58d57b5a9e35-kube-api-access-hkkx9\") pod \"perf-node-gather-daemonset-kmsxc\" (UID: \"aae40325-6969-414a-b615-58d57b5a9e35\") " pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-kmsxc" Oct 30 23:05:15.313403 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:15.313364 6006 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-kmsxc" Oct 30 23:05:15.683890 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:15.683861 6006 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2cfz8/perf-node-gather-daemonset-kmsxc"] Oct 30 23:05:15.686788 ip-10-0-135-7 kubenswrapper[6006]: W1030 23:05:15.686755 6006 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaae40325_6969_414a_b615_58d57b5a9e35.slice/crio-a682150c3f8225086b8e950e9a131e3436737752f6a9877a24d9ca787087532a WatchSource:0}: Error finding container a682150c3f8225086b8e950e9a131e3436737752f6a9877a24d9ca787087532a: Status 404 returned error can't find the container with id a682150c3f8225086b8e950e9a131e3436737752f6a9877a24d9ca787087532a Oct 30 23:05:16.632317 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:16.632286 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-p76jm_47477b2f-5e3f-437c-8203-80aac978e56a/limitador/0.log" Oct 30 23:05:16.685727 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:16.685691 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-kmsxc" event={"ID":"aae40325-6969-414a-b615-58d57b5a9e35","Type":"ContainerStarted","Data":"ea065ff4c2272cc2218cb0dd7deb553c6f8d14cb7559287db36c1c86ddb6b96b"} Oct 30 23:05:16.686127 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:16.685750 6006 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-kmsxc" event={"ID":"aae40325-6969-414a-b615-58d57b5a9e35","Type":"ContainerStarted","Data":"a682150c3f8225086b8e950e9a131e3436737752f6a9877a24d9ca787087532a"} Oct 30 23:05:16.686127 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:16.685868 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-kmsxc" Oct 30 23:05:16.803730 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:16.797300 6006 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-kmsxc" podStartSLOduration=2.797275232 podStartE2EDuration="2.797275232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:16.794852614 +0000 UTC m=+4519.347028914" watchObservedRunningTime="2026-10-30 23:05:16.797275232 +0000 UTC m=+4519.349451533" Oct 30 23:05:17.087600 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:17.087559 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-7gcxz_e7333812-2624-427a-ac27-7f953e3d517e/manager/0.log" Oct 30 23:05:22.489658 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:22.489627 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-hmttl_95097323-8907-45bc-8967-0de718adefbd/cluster-monitoring-operator/0.log" Oct 30 23:05:22.701622 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:22.701591 6006 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2cfz8/perf-node-gather-daemonset-kmsxc" Oct 30 23:05:27.506218 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:27.506184 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wprgj_94550423-24fa-4ec9-b44d-9cd54b08f75d/node-exporter/0.log" Oct 30 23:05:27.910447 ip-10-0-135-7 kubenswrapper[6006]: E1030 23:05:27.910405 6006 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hkqxb" containerName="extractor" Oct 30 23:05:27.917737 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:27.917710 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wprgj_94550423-24fa-4ec9-b44d-9cd54b08f75d/kube-rbac-proxy/0.log" Oct 30 23:05:28.327532 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:28.327502 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wprgj_94550423-24fa-4ec9-b44d-9cd54b08f75d/init-textfile/0.log" Oct 30 23:05:33.634279 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:33.634238 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-bc452_8df7e7fd-be92-4d37-af33-930b7b36cbe6/prometheus-operator-admission-webhook/0.log" Oct 30 23:05:37.212119 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:37.212082 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-65844c896fp8nft_e592cd54-dc62-473d-9993-976186532144/istio-proxy/0.log" Oct 30 23:05:38.045181 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:38.045149 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-kv5vn_64dd55d9-83da-4513-b49a-fbca9fc1cd9b/discovery/0.log" Oct 30 23:05:38.778089 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:38.778052 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-6cmps_2c3b94ab-018e-480f-b12c-faa5544fc2e1/istio-proxy/0.log" Oct 30 23:05:39.200100 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:39.200059 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-ck8bj_33b53342-a5ab-459b-8005-da1267a7c0b4/istio-proxy/0.log" Oct 30 23:05:39.618768 ip-10-0-135-7 kubenswrapper[6006]: I1030 23:05:39.618735 6006 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-jst4g_82a12523-b466-4041-bb2a-73db5538706c/istio-proxy/0.log"